[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 01:52:21.560737 20741 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.65.126:35273,127.20.65.125:32799,127.20.65.124:42919
I20240502 01:52:21.562440 20741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:21.563525 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:21.575636 20747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:21.576277 20746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:21.576877 20750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:21.577652 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:21.578384 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:21.578577 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:21.578663 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614741578647 us; error 0 us; skew 500 ppm
I20240502 01:52:21.578927 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:21.586836 20741 webserver.cc:460] Webserver started at http://127.20.65.126:46779/ using document root <none> and password file <none>
I20240502 01:52:21.587514 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:21.587600 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:21.587924 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:21.591198 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-0-root/instance:
uuid: "868791bff2474c99a9cb621286399803"
format_stamp: "Formatted at 2024-05-02 01:52:21 on dist-test-slave-7cj2"
I20240502 01:52:21.596809 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:52:21.600044 20755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:21.601347 20741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 01:52:21.601523 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-0-root
uuid: "868791bff2474c99a9cb621286399803"
format_stamp: "Formatted at 2024-05-02 01:52:21 on dist-test-slave-7cj2"
I20240502 01:52:21.601678 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:21.628806 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:21.629864 20741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:21.630209 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:21.656400 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.126:35273
I20240502 01:52:21.656433 20806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.126:35273 every 8 connection(s)
I20240502 01:52:21.658753 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:21.662026 20809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:21.662370 20807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:52:21.662935 20810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:21.663493 20812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:21.664104 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:21.664425 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:21.664496 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:21.664588 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614741664534 us; error 0 us; skew 500 ppm
I20240502 01:52:21.664764 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:21.666163 20741 webserver.cc:460] Webserver started at http://127.20.65.125:40751/ using document root <none> and password file <none>
I20240502 01:52:21.666380 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:21.666497 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:21.666656 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:21.667279 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-1-root/instance:
uuid: "6503046d389441738f85a3df151cf8be"
format_stamp: "Formatted at 2024-05-02 01:52:21 on dist-test-slave-7cj2"
I20240502 01:52:21.669934 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 01:52:21.671753 20817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:21.672120 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:52:21.672240 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-1-root
uuid: "6503046d389441738f85a3df151cf8be"
format_stamp: "Formatted at 2024-05-02 01:52:21 on dist-test-slave-7cj2"
I20240502 01:52:21.672389 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:21.674533 20807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:52:21.693287 20807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } has no permanent_uuid. Determining permanent_uuid...
W20240502 01:52:21.694173 20756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.65.125:32799: connect: Connection refused (error 111)
W20240502 01:52:21.695734 20807 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.65.125:32799: Network error: Client connection negotiation failed: client connection to 127.20.65.125:32799: connect: Connection refused (error 111)
I20240502 01:52:21.700915 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:21.701501 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:21.716734 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.125:32799
I20240502 01:52:21.716778 20871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.125:32799 every 8 connection(s)
I20240502 01:52:21.718190 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:52:21.718261 20872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:21.721975 20872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } has no permanent_uuid. Determining permanent_uuid...
W20240502 01:52:21.722236 20875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:21.722852 20874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:21.724952 20877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:21.725566 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:21.725849 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:21.725924 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:21.726009 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614741725968 us; error 0 us; skew 500 ppm
I20240502 01:52:21.726168 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:21.727854 20741 webserver.cc:460] Webserver started at http://127.20.65.124:42937/ using document root <none> and password file <none>
I20240502 01:52:21.728161 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:21.728256 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:21.728410 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:21.729233 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-2-root/instance:
uuid: "a90a6821365c4a0f9b30d25cb394210a"
format_stamp: "Formatted at 2024-05-02 01:52:21 on dist-test-slave-7cj2"
I20240502 01:52:21.729812 20872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:52:21.732475 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:52:21.734438 20885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:21.734908 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 01:52:21.735051 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-2-root
uuid: "a90a6821365c4a0f9b30d25cb394210a"
format_stamp: "Formatted at 2024-05-02 01:52:21 on dist-test-slave-7cj2"
I20240502 01:52:21.735232 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:21.736905 20872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } has no permanent_uuid. Determining permanent_uuid...
W20240502 01:52:21.739197 20872 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.65.124:42919: Network error: Client connection negotiation failed: client connection to 127.20.65.124:42919: connect: Connection refused (error 111)
I20240502 01:52:21.745030 20807 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } attempt: 1
I20240502 01:52:21.747890 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:21.748581 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:21.751183 20807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } has no permanent_uuid. Determining permanent_uuid...
W20240502 01:52:21.752892 20807 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.65.124:42919: Network error: Client connection negotiation failed: client connection to 127.20.65.124:42919: connect: Connection refused (error 111)
I20240502 01:52:21.763831 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.124:42919
I20240502 01:52:21.763900 20937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.124:42919 every 8 connection(s)
I20240502 01:52:21.764896 20741 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:52:21.765257 20938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:21.767594 20938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:52:21.773795 20938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:52:21.778587 20938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:52:21.787930 20938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a: Bootstrap starting.
I20240502 01:52:21.791311 20938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:21.791522 20872 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } attempt: 1
I20240502 01:52:21.792979 20938 log.cc:824] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:21.796067 20807 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } attempt: 1
I20240502 01:52:21.796458 20938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a: No bootstrap required, opened a new log
I20240502 01:52:21.801679 20872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be: Bootstrap starting.
I20240502 01:52:21.803604 20872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:21.805235 20807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803: Bootstrap starting.
I20240502 01:52:21.805785 20872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be: No bootstrap required, opened a new log
I20240502 01:52:21.806663 20807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:21.807251 20938 raft_consensus.cc:348] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } }
I20240502 01:52:21.807250 20872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } }
I20240502 01:52:21.807685 20938 raft_consensus.cc:374] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:21.807701 20872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:21.807859 20938 raft_consensus.cc:729] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90a6821365c4a0f9b30d25cb394210a, State: Initialized, Role: FOLLOWER
I20240502 01:52:21.807857 20872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503046d389441738f85a3df151cf8be, State: Initialized, Role: FOLLOWER
I20240502 01:52:21.808696 20807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803: No bootstrap required, opened a new log
I20240502 01:52:21.808696 20938 consensus_queue.cc:260] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } }
I20240502 01:52:21.808718 20872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } }
I20240502 01:52:21.809638 20807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } }
I20240502 01:52:21.809945 20807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:21.810058 20807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868791bff2474c99a9cb621286399803, State: Initialized, Role: FOLLOWER
I20240502 01:52:21.809798 20946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } } }
I20240502 01:52:21.810330 20946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:52:21.810437 20807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } }
I20240502 01:52:21.811245 20872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:21.811290 20949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } } }
I20240502 01:52:21.811712 20807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:21.811735 20949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:52:21.818738 20948 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } } }
I20240502 01:52:21.819002 20938 sys_catalog.cc:564] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:21.819032 20948 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:52:21.831272 20741 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240502 01:52:21.836566 20741 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240502 01:52:21.837512 20980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:52:21.838083 20983 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240502 01:52:21.837746 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:21.838201 20983 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 01:52:21.838299 20980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 01:52:21.838076 20981 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:52:21.838454 20981 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 01:52:21.842702 20984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:21.843164 20985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:21.843758 20987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:21.847585 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:21.847867 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:21.847968 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:21.848035 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614741848009 us; error 0 us; skew 500 ppm
I20240502 01:52:21.848232 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:21.850786 20741 webserver.cc:460] Webserver started at http://127.20.65.65:35769/ using document root <none> and password file <none>
I20240502 01:52:21.851653 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:21.851938 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:21.852298 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:21.853199 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-0-root/instance:
uuid: "9d02782ebc4b473c91ccf520ff3e80b5"
format_stamp: "Formatted at 2024-05-02 01:52:21 on dist-test-slave-7cj2"
I20240502 01:52:21.856850 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240502 01:52:21.859063 20992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:21.859455 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.001s
I20240502 01:52:21.859561 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-0-root
uuid: "9d02782ebc4b473c91ccf520ff3e80b5"
format_stamp: "Formatted at 2024-05-02 01:52:21 on dist-test-slave-7cj2"
I20240502 01:52:21.859696 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240502 01:52:21.882956 20761 tablet.cc:2369] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:21.893597 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:21.894225 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:21.895090 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:21.896593 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:21.896737 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:21.896857 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:21.896939 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:21.917893 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.65:43363
I20240502 01:52:21.917976 21055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.65:43363 every 8 connection(s)
I20240502 01:52:21.935811 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:52:21.939167 21057 heartbeater.cc:347] Connected to a master server at 127.20.65.124:42919
I20240502 01:52:21.939528 21057 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:21.940073 21056 heartbeater.cc:347] Connected to a master server at 127.20.65.126:35273
I20240502 01:52:21.940201 21056 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:21.941102 21057 heartbeater.cc:511] Master 127.20.65.124:42919 requested a full tablet report, sending...
I20240502 01:52:21.944279 20903 ts_manager.cc:194] Registered new tserver with Master: 9d02782ebc4b473c91ccf520ff3e80b5 (127.20.65.65:43363)
I20240502 01:52:21.945724 21056 heartbeater.cc:511] Master 127.20.65.126:35273 requested a full tablet report, sending...
I20240502 01:52:21.947177 20772 ts_manager.cc:194] Registered new tserver with Master: 9d02782ebc4b473c91ccf520ff3e80b5 (127.20.65.65:43363)
W20240502 01:52:21.947209 21064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:21.948979 21063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:21.949539 21066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:21.952980 21058 heartbeater.cc:347] Connected to a master server at 127.20.65.125:32799
I20240502 01:52:21.953151 21058 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:21.953661 21058 heartbeater.cc:511] Master 127.20.65.125:32799 requested a full tablet report, sending...
I20240502 01:52:21.954212 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:21.954573 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:21.954679 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:21.954773 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614741954723 us; error 0 us; skew 500 ppm
I20240502 01:52:21.954931 20837 ts_manager.cc:194] Registered new tserver with Master: 9d02782ebc4b473c91ccf520ff3e80b5 (127.20.65.65:43363)
I20240502 01:52:21.954972 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:21.956622 20741 webserver.cc:460] Webserver started at http://127.20.65.66:34645/ using document root <none> and password file <none>
I20240502 01:52:21.956905 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:21.957022 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:21.957180 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:21.958041 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-1-root/instance:
uuid: "005d65c95f48465095f3a07084a7a324"
format_stamp: "Formatted at 2024-05-02 01:52:21 on dist-test-slave-7cj2"
I20240502 01:52:21.961551 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:52:21.963716 21071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:21.964133 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:52:21.964234 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-1-root
uuid: "005d65c95f48465095f3a07084a7a324"
format_stamp: "Formatted at 2024-05-02 01:52:21 on dist-test-slave-7cj2"
I20240502 01:52:21.964380 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:21.986119 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:21.986730 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:21.987303 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:21.988258 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:21.988358 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:21.988467 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:21.988526 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:22.014293 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.66:38071
I20240502 01:52:22.014293 21133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.66:38071 every 8 connection(s)
I20240502 01:52:22.036978 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:52:22.041508 21134 heartbeater.cc:347] Connected to a master server at 127.20.65.126:35273
I20240502 01:52:22.041744 21134 heartbeater.cc:464] Registering TS with master...
W20240502 01:52:22.042610 21140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:22.043269 21134 heartbeater.cc:511] Master 127.20.65.126:35273 requested a full tablet report, sending...
I20240502 01:52:22.044783 20772 ts_manager.cc:194] Registered new tserver with Master: 005d65c95f48465095f3a07084a7a324 (127.20.65.66:38071)
I20240502 01:52:22.045235 21135 heartbeater.cc:347] Connected to a master server at 127.20.65.124:42919
I20240502 01:52:22.045396 21135 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:22.046293 21135 heartbeater.cc:511] Master 127.20.65.124:42919 requested a full tablet report, sending...
W20240502 01:52:22.047456 21141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:22.047508 20903 ts_manager.cc:194] Registered new tserver with Master: 005d65c95f48465095f3a07084a7a324 (127.20.65.66:38071)
I20240502 01:52:22.053402 20741 server_base.cc:1034] running on GCE node
W20240502 01:52:22.054020 21143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:22.054144 21136 heartbeater.cc:347] Connected to a master server at 127.20.65.125:32799
I20240502 01:52:22.054301 21136 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:22.054637 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:22.054769 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:22.054865 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614742054819 us; error 0 us; skew 500 ppm
I20240502 01:52:22.054870 21136 heartbeater.cc:511] Master 127.20.65.125:32799 requested a full tablet report, sending...
I20240502 01:52:22.055080 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:22.056298 20837 ts_manager.cc:194] Registered new tserver with Master: 005d65c95f48465095f3a07084a7a324 (127.20.65.66:38071)
I20240502 01:52:22.058684 20741 webserver.cc:460] Webserver started at http://127.20.65.67:39325/ using document root <none> and password file <none>
I20240502 01:52:22.058909 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:22.058998 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:22.059119 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:22.059852 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-2-root/instance:
uuid: "c0cb315df19d45bd9913e6f2cae57fc7"
format_stamp: "Formatted at 2024-05-02 01:52:22 on dist-test-slave-7cj2"
I20240502 01:52:22.062916 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:52:22.064791 21148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:22.065225 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 01:52:22.065351 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-2-root
uuid: "c0cb315df19d45bd9913e6f2cae57fc7"
format_stamp: "Formatted at 2024-05-02 01:52:22 on dist-test-slave-7cj2"
I20240502 01:52:22.065479 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:22.068496 20948 raft_consensus.cc:482] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:22.068789 20948 raft_consensus.cc:504] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } }
I20240502 01:52:22.070696 20948 leader_election.cc:288] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 868791bff2474c99a9cb621286399803 (127.20.65.126:35273), 6503046d389441738f85a3df151cf8be (127.20.65.125:32799)
I20240502 01:52:22.071528 20782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a90a6821365c4a0f9b30d25cb394210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868791bff2474c99a9cb621286399803" is_pre_election: true
I20240502 01:52:22.071578 20847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a90a6821365c4a0f9b30d25cb394210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503046d389441738f85a3df151cf8be" is_pre_election: true
I20240502 01:52:22.071941 20782 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a90a6821365c4a0f9b30d25cb394210a in term 0.
I20240502 01:52:22.071949 20847 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a90a6821365c4a0f9b30d25cb394210a in term 0.
I20240502 01:52:22.072615 20886 leader_election.cc:302] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 868791bff2474c99a9cb621286399803, a90a6821365c4a0f9b30d25cb394210a; no voters:
I20240502 01:52:22.073122 20948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:22.073257 20948 raft_consensus.cc:482] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:22.073345 20948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:22.076581 20948 raft_consensus.cc:504] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } }
I20240502 01:52:22.077214 20948 leader_election.cc:288] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [CANDIDATE]: Term 1 election: Requested vote from peers 868791bff2474c99a9cb621286399803 (127.20.65.126:35273), 6503046d389441738f85a3df151cf8be (127.20.65.125:32799)
I20240502 01:52:22.077699 20847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a90a6821365c4a0f9b30d25cb394210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6503046d389441738f85a3df151cf8be"
I20240502 01:52:22.077872 20847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:22.077778 20782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a90a6821365c4a0f9b30d25cb394210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868791bff2474c99a9cb621286399803"
I20240502 01:52:22.078141 20782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:22.081321 20847 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a90a6821365c4a0f9b30d25cb394210a in term 1.
I20240502 01:52:22.081344 20782 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a90a6821365c4a0f9b30d25cb394210a in term 1.
I20240502 01:52:22.081791 20886 leader_election.cc:302] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 868791bff2474c99a9cb621286399803, a90a6821365c4a0f9b30d25cb394210a; no voters:
I20240502 01:52:22.082096 20948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:22.082682 20948 raft_consensus.cc:686] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [term 1 LEADER]: Becoming Leader. State: Replica: a90a6821365c4a0f9b30d25cb394210a, State: Running, Role: LEADER
I20240502 01:52:22.083377 20948 consensus_queue.cc:237] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } }
I20240502 01:52:22.087082 21153 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a90a6821365c4a0f9b30d25cb394210a. Latest consensus state: current_term: 1 leader_uuid: "a90a6821365c4a0f9b30d25cb394210a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } } }
I20240502 01:52:22.087399 21153 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:22.088932 21155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:22.090818 21155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:22.098539 20847 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [term 1 FOLLOWER]: Refusing update from remote peer a90a6821365c4a0f9b30d25cb394210a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:22.098539 20782 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [term 1 FOLLOWER]: Refusing update from remote peer a90a6821365c4a0f9b30d25cb394210a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:22.099530 20948 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [LEADER]: Connected to new peer: Peer: permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:22.100250 21153 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:22.117725 20949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a90a6821365c4a0f9b30d25cb394210a. Latest consensus state: current_term: 1 leader_uuid: "a90a6821365c4a0f9b30d25cb394210a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } } }
I20240502 01:52:22.118166 20949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:52:22.122974 21157 mvcc.cc:205] Tried to move back new op lower bound from 7023061983625551872 to 7023061983578976256. Current Snapshot: MvccSnapshot[applied={T|T < 7023061983625551872}]
I20240502 01:52:22.131987 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:22.132855 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:22.134048 20949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a90a6821365c4a0f9b30d25cb394210a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } } }
I20240502 01:52:22.134371 20949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:52:22.135862 21153 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a90a6821365c4a0f9b30d25cb394210a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } } }
I20240502 01:52:22.136132 21153 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:22.137353 21155 catalog_manager.cc:1347] Generated new cluster ID: 12e6f6b856574987a4d3438b8b8b1186
I20240502 01:52:22.137512 21155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:22.139575 20946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [sys.catalog]: SysCatalogTable state changed. Reason: New leader a90a6821365c4a0f9b30d25cb394210a. Latest consensus state: current_term: 1 leader_uuid: "a90a6821365c4a0f9b30d25cb394210a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } } }
I20240502 01:52:22.139782 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:22.139880 20946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:52:22.141445 20948 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a90a6821365c4a0f9b30d25cb394210a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } } }
I20240502 01:52:22.141719 20948 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:22.147186 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:22.147293 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:22.147397 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:22.147475 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:22.159018 20946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a90a6821365c4a0f9b30d25cb394210a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868791bff2474c99a9cb621286399803" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 35273 } } peers { permanent_uuid: "6503046d389441738f85a3df151cf8be" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 32799 } } peers { permanent_uuid: "a90a6821365c4a0f9b30d25cb394210a" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 42919 } } }
I20240502 01:52:22.159287 20946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:52:22.188949 21155 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:22.190224 21155 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:22.214452 21155 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a: Generated new TSK 0
I20240502 01:52:22.215674 21155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:22.219614 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.67:35831
I20240502 01:52:22.219745 21222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.67:35831 every 8 connection(s)
I20240502 01:52:22.251554 21226 heartbeater.cc:347] Connected to a master server at 127.20.65.125:32799
I20240502 01:52:22.251766 21226 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:22.252548 21226 heartbeater.cc:511] Master 127.20.65.125:32799 requested a full tablet report, sending...
I20240502 01:52:22.254101 20837 ts_manager.cc:194] Registered new tserver with Master: c0cb315df19d45bd9913e6f2cae57fc7 (127.20.65.67:35831)
I20240502 01:52:22.254719 21223 heartbeater.cc:347] Connected to a master server at 127.20.65.126:35273
I20240502 01:52:22.254879 21223 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:22.255561 21223 heartbeater.cc:511] Master 127.20.65.126:35273 requested a full tablet report, sending...
I20240502 01:52:22.256795 20772 ts_manager.cc:194] Registered new tserver with Master: c0cb315df19d45bd9913e6f2cae57fc7 (127.20.65.67:35831)
I20240502 01:52:22.257818 21224 heartbeater.cc:347] Connected to a master server at 127.20.65.124:42919
I20240502 01:52:22.257942 21224 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:22.258510 21224 heartbeater.cc:511] Master 127.20.65.124:42919 requested a full tablet report, sending...
I20240502 01:52:22.259379 20903 ts_manager.cc:194] Registered new tserver with Master: c0cb315df19d45bd9913e6f2cae57fc7 (127.20.65.67:35831)
I20240502 01:52:22.260378 20741 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003685079s
I20240502 01:52:22.261595 20903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48058
I20240502 01:52:22.840159 20980 catalog_manager.cc:1259] Loaded cluster ID: 12e6f6b856574987a4d3438b8b8b1186
I20240502 01:52:22.840193 20983 catalog_manager.cc:1259] Loaded cluster ID: 12e6f6b856574987a4d3438b8b8b1186
I20240502 01:52:22.840303 20980 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803: loading cluster ID for follower catalog manager: success
I20240502 01:52:22.840303 20983 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be: loading cluster ID for follower catalog manager: success
I20240502 01:52:22.842634 20980 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803: acquiring CA information for follower catalog manager: success
I20240502 01:52:22.842680 20983 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be: acquiring CA information for follower catalog manager: success
I20240502 01:52:22.844002 20983 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 01:52:22.843998 20980 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 01:52:22.947633 20903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48034
I20240502 01:52:23.051679 20903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48044
I20240502 01:52:23.263767 21224 heartbeater.cc:503] Master 127.20.65.124:42919 was elected leader, sending a full tablet report...
I20240502 01:52:23.832988 20974 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:52:23.949215 21057 heartbeater.cc:503] Master 127.20.65.124:42919 was elected leader, sending a full tablet report...
I20240502 01:52:24.052855 21135 heartbeater.cc:503] Master 127.20.65.124:42919 was elected leader, sending a full tablet report...
I20240502 01:52:24.321883 20741 test_util.cc:268] Using random seed: -624750796
I20240502 01:52:24.352979 20903 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48062:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 01:52:24.393376 21099 tablet_service.cc:1467] Processing CreateTablet for tablet 914c97ae25e34bcda35de5e65c15d1bc (DEFAULT_TABLE table=test-workload [id=127a0c72bb2e494fa33e2bc5a98ea915]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 01:52:24.393421 21187 tablet_service.cc:1467] Processing CreateTablet for tablet 77c6376566ec4a688be9a40e28b36af2 (DEFAULT_TABLE table=test-workload [id=127a0c72bb2e494fa33e2bc5a98ea915]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 01:52:24.394291 21099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 914c97ae25e34bcda35de5e65c15d1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:24.394346 21187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c6376566ec4a688be9a40e28b36af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:24.396566 21021 tablet_service.cc:1467] Processing CreateTablet for tablet 711f5f79990e4964b189ea4dc5cb5a02 (DEFAULT_TABLE table=test-workload [id=127a0c72bb2e494fa33e2bc5a98ea915]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 01:52:24.396622 21020 tablet_service.cc:1467] Processing CreateTablet for tablet a4a49020d83e489a9d9946ccc095c79b (DEFAULT_TABLE table=test-workload [id=127a0c72bb2e494fa33e2bc5a98ea915]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 01:52:24.397150 21021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711f5f79990e4964b189ea4dc5cb5a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:24.397451 21020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a49020d83e489a9d9946ccc095c79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:24.406888 21258 tablet_bootstrap.cc:492] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7: Bootstrap starting.
I20240502 01:52:24.407132 21259 tablet_bootstrap.cc:492] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324: Bootstrap starting.
I20240502 01:52:24.407613 21260 tablet_bootstrap.cc:492] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5: Bootstrap starting.
I20240502 01:52:24.409498 21260 tablet_bootstrap.cc:654] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:24.410066 21259 tablet_bootstrap.cc:654] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:24.410221 21258 tablet_bootstrap.cc:654] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:24.412110 21260 tablet_bootstrap.cc:492] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5: No bootstrap required, opened a new log
I20240502 01:52:24.412235 21260 ts_tablet_manager.cc:1367] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 01:52:24.412993 21260 raft_consensus.cc:348] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d02782ebc4b473c91ccf520ff3e80b5" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43363 } }
I20240502 01:52:24.413139 21260 raft_consensus.cc:374] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:24.413239 21260 raft_consensus.cc:729] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d02782ebc4b473c91ccf520ff3e80b5, State: Initialized, Role: FOLLOWER
I20240502 01:52:24.413493 21260 consensus_queue.cc:260] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d02782ebc4b473c91ccf520ff3e80b5" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43363 } }
I20240502 01:52:24.413713 21260 raft_consensus.cc:388] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:24.413810 21260 raft_consensus.cc:482] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:24.413916 21260 raft_consensus.cc:3037] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:24.415418 21258 tablet_bootstrap.cc:492] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7: No bootstrap required, opened a new log
I20240502 01:52:24.415474 21259 tablet_bootstrap.cc:492] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324: No bootstrap required, opened a new log
I20240502 01:52:24.415674 21259 ts_tablet_manager.cc:1367] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20240502 01:52:24.415670 21258 ts_tablet_manager.cc:1367] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 01:52:24.416355 21260 raft_consensus.cc:504] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d02782ebc4b473c91ccf520ff3e80b5" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43363 } }
I20240502 01:52:24.416596 21260 leader_election.cc:302] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d02782ebc4b473c91ccf520ff3e80b5; no voters:
I20240502 01:52:24.416574 21259 raft_consensus.cc:348] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005d65c95f48465095f3a07084a7a324" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 38071 } }
I20240502 01:52:24.416543 21258 raft_consensus.cc:348] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cb315df19d45bd9913e6f2cae57fc7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 35831 } }
I20240502 01:52:24.416749 21259 raft_consensus.cc:374] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:24.416831 21258 raft_consensus.cc:374] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:24.416880 21259 raft_consensus.cc:729] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 005d65c95f48465095f3a07084a7a324, State: Initialized, Role: FOLLOWER
I20240502 01:52:24.416944 21258 raft_consensus.cc:729] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0cb315df19d45bd9913e6f2cae57fc7, State: Initialized, Role: FOLLOWER
I20240502 01:52:24.417376 21260 leader_election.cc:288] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:24.417300 21258 consensus_queue.cc:260] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cb315df19d45bd9913e6f2cae57fc7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 35831 } }
I20240502 01:52:24.417289 21259 consensus_queue.cc:260] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005d65c95f48465095f3a07084a7a324" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 38071 } }
I20240502 01:52:24.417579 21264 raft_consensus.cc:2781] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:24.417626 21259 raft_consensus.cc:388] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:24.417724 21259 raft_consensus.cc:482] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:24.417877 21259 raft_consensus.cc:3037] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:24.418555 21260 ts_tablet_manager.cc:1398] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20240502 01:52:24.419507 21260 tablet_bootstrap.cc:492] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5: Bootstrap starting.
I20240502 01:52:24.420349 21259 raft_consensus.cc:504] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005d65c95f48465095f3a07084a7a324" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 38071 } }
I20240502 01:52:24.420635 21259 leader_election.cc:302] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 005d65c95f48465095f3a07084a7a324; no voters:
I20240502 01:52:24.421125 21258 raft_consensus.cc:388] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:24.421244 21258 raft_consensus.cc:482] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:24.421376 21258 raft_consensus.cc:3037] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:24.422109 21260 tablet_bootstrap.cc:654] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:24.424396 21264 raft_consensus.cc:686] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5 [term 1 LEADER]: Becoming Leader. State: Replica: 9d02782ebc4b473c91ccf520ff3e80b5, State: Running, Role: LEADER
I20240502 01:52:24.424672 21259 leader_election.cc:288] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:24.424628 21258 raft_consensus.cc:504] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cb315df19d45bd9913e6f2cae57fc7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 35831 } }
I20240502 01:52:24.424824 21267 raft_consensus.cc:2781] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:24.425014 21258 leader_election.cc:302] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0cb315df19d45bd9913e6f2cae57fc7; no voters:
I20240502 01:52:24.425773 21259 ts_tablet_manager.cc:1398] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20240502 01:52:24.424979 21264 consensus_queue.cc:237] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d02782ebc4b473c91ccf520ff3e80b5" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43363 } }
I20240502 01:52:24.428130 21267 raft_consensus.cc:686] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324 [term 1 LEADER]: Becoming Leader. State: Replica: 005d65c95f48465095f3a07084a7a324, State: Running, Role: LEADER
I20240502 01:52:24.428469 21258 leader_election.cc:288] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:24.428927 21267 consensus_queue.cc:237] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005d65c95f48465095f3a07084a7a324" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 38071 } }
I20240502 01:52:24.433854 20903 catalog_manager.cc:5484] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d02782ebc4b473c91ccf520ff3e80b5 (127.20.65.65). New cstate: current_term: 1 leader_uuid: "9d02782ebc4b473c91ccf520ff3e80b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d02782ebc4b473c91ccf520ff3e80b5" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43363 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:24.435078 21270 raft_consensus.cc:2781] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:24.440742 21260 tablet_bootstrap.cc:492] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5: No bootstrap required, opened a new log
I20240502 01:52:24.440940 21260 ts_tablet_manager.cc:1367] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.004s
I20240502 01:52:24.441900 21260 raft_consensus.cc:348] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d02782ebc4b473c91ccf520ff3e80b5" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43363 } }
I20240502 01:52:24.442147 21260 raft_consensus.cc:374] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:24.442245 21260 raft_consensus.cc:729] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d02782ebc4b473c91ccf520ff3e80b5, State: Initialized, Role: FOLLOWER
I20240502 01:52:24.443841 21260 consensus_queue.cc:260] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d02782ebc4b473c91ccf520ff3e80b5" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43363 } }
I20240502 01:52:24.444084 21260 raft_consensus.cc:388] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:24.444242 21260 raft_consensus.cc:482] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:24.444362 21260 raft_consensus.cc:3037] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:24.444765 21270 raft_consensus.cc:686] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7 [term 1 LEADER]: Becoming Leader. State: Replica: c0cb315df19d45bd9913e6f2cae57fc7, State: Running, Role: LEADER
I20240502 01:52:24.445111 21270 consensus_queue.cc:237] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cb315df19d45bd9913e6f2cae57fc7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 35831 } }
I20240502 01:52:24.447494 21260 raft_consensus.cc:504] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d02782ebc4b473c91ccf520ff3e80b5" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43363 } }
I20240502 01:52:24.447769 21260 leader_election.cc:302] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d02782ebc4b473c91ccf520ff3e80b5; no voters:
I20240502 01:52:24.448009 21260 leader_election.cc:288] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:24.448246 21265 raft_consensus.cc:2781] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:24.449959 20903 catalog_manager.cc:5484] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0cb315df19d45bd9913e6f2cae57fc7 (127.20.65.67). New cstate: current_term: 1 leader_uuid: "c0cb315df19d45bd9913e6f2cae57fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cb315df19d45bd9913e6f2cae57fc7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 35831 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:24.456825 21260 ts_tablet_manager.cc:1398] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20240502 01:52:24.468787 21258 ts_tablet_manager.cc:1398] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7: Time spent starting tablet: real 0.052s user 0.008s sys 0.014s
I20240502 01:52:24.471920 21265 raft_consensus.cc:686] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5 [term 1 LEADER]: Becoming Leader. State: Replica: 9d02782ebc4b473c91ccf520ff3e80b5, State: Running, Role: LEADER
I20240502 01:52:24.473348 21265 consensus_queue.cc:237] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d02782ebc4b473c91ccf520ff3e80b5" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43363 } }
I20240502 01:52:24.473352 20902 catalog_manager.cc:5484] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324 reported cstate change: term changed from 0 to 1, leader changed from <none> to 005d65c95f48465095f3a07084a7a324 (127.20.65.66). New cstate: current_term: 1 leader_uuid: "005d65c95f48465095f3a07084a7a324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005d65c95f48465095f3a07084a7a324" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 38071 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:24.497967 20901 catalog_manager.cc:5484] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d02782ebc4b473c91ccf520ff3e80b5 (127.20.65.65). New cstate: current_term: 1 leader_uuid: "9d02782ebc4b473c91ccf520ff3e80b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d02782ebc4b473c91ccf520ff3e80b5" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43363 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:24.526120 20741 tablet_server.cc:178] TabletServer@127.20.65.65:0 shutting down...
I20240502 01:52:24.535938 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:52:24.536410 20741 tablet_replica.cc:330] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5: stopping tablet replica
I20240502 01:52:24.536868 20741 raft_consensus.cc:2229] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:52:24.537106 20741 raft_consensus.cc:2258] T a4a49020d83e489a9d9946ccc095c79b P 9d02782ebc4b473c91ccf520ff3e80b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:24.538544 20741 tablet_replica.cc:330] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5: stopping tablet replica
I20240502 01:52:24.538677 20741 raft_consensus.cc:2229] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:52:24.538802 20741 raft_consensus.cc:2258] T 711f5f79990e4964b189ea4dc5cb5a02 P 9d02782ebc4b473c91ccf520ff3e80b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:24.544183 20741 tablet_server.cc:194] TabletServer@127.20.65.65:0 shutdown complete.
I20240502 01:52:24.549048 20741 tablet_server.cc:178] TabletServer@127.20.65.66:0 shutting down...
I20240502 01:52:24.557147 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:52:24.557490 20741 tablet_replica.cc:330] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324: stopping tablet replica
I20240502 01:52:24.557667 20741 raft_consensus.cc:2229] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:52:24.557806 20741 raft_consensus.cc:2258] T 914c97ae25e34bcda35de5e65c15d1bc P 005d65c95f48465095f3a07084a7a324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:24.571620 20741 tablet_server.cc:194] TabletServer@127.20.65.66:0 shutdown complete.
I20240502 01:52:24.574275 20741 tablet_server.cc:178] TabletServer@127.20.65.67:0 shutting down...
I20240502 01:52:24.584906 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:52:24.585261 20741 tablet_replica.cc:330] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7: stopping tablet replica
I20240502 01:52:24.585441 20741 raft_consensus.cc:2229] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:52:24.585634 20741 raft_consensus.cc:2258] T 77c6376566ec4a688be9a40e28b36af2 P c0cb315df19d45bd9913e6f2cae57fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:24.589761 20741 tablet_server.cc:194] TabletServer@127.20.65.67:0 shutdown complete.
I20240502 01:52:24.592487 20741 master.cc:537] Master@127.20.65.126:35273 shutting down...
I20240502 01:52:24.601066 20741 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:24.601294 20741 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:24.601441 20741 tablet_replica.cc:330] T 00000000000000000000000000000000 P 868791bff2474c99a9cb621286399803: stopping tablet replica
I20240502 01:52:24.615532 20741 master.cc:552] Master@127.20.65.126:35273 shutdown complete.
I20240502 01:52:24.620064 20741 master.cc:537] Master@127.20.65.125:32799 shutting down...
I20240502 01:52:24.626925 20741 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:24.627137 20741 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:24.627249 20741 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6503046d389441738f85a3df151cf8be: stopping tablet replica
I20240502 01:52:24.641108 20741 master.cc:552] Master@127.20.65.125:32799 shutdown complete.
I20240502 01:52:24.645690 20741 master.cc:537] Master@127.20.65.124:42919 shutting down...
I20240502 01:52:24.652871 20741 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:52:24.653301 20741 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:24.653442 20741 tablet_replica.cc:330] T 00000000000000000000000000000000 P a90a6821365c4a0f9b30d25cb394210a: stopping tablet replica
I20240502 01:52:24.667940 20741 master.cc:552] Master@127.20.65.124:42919 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (3123 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240502 01:52:24.682165 20741 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.65.126:34393
I20240502 01:52:24.682703 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:24.685483 21286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:24.686302 21287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:24.686977 21289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:24.687839 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:24.688071 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:24.688154 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:24.688211 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614744688195 us; error 0 us; skew 500 ppm
I20240502 01:52:24.688392 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:24.689625 20741 webserver.cc:460] Webserver started at http://127.20.65.126:33565/ using document root <none> and password file <none>
I20240502 01:52:24.689862 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:24.689941 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:24.690093 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:24.690786 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/master-0-root/instance:
uuid: "7652667f8dda40d8b6bdba0930be5092"
format_stamp: "Formatted at 2024-05-02 01:52:24 on dist-test-slave-7cj2"
I20240502 01:52:24.693374 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:52:24.695003 21294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:24.695389 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:52:24.695482 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/master-0-root
uuid: "7652667f8dda40d8b6bdba0930be5092"
format_stamp: "Formatted at 2024-05-02 01:52:24 on dist-test-slave-7cj2"
I20240502 01:52:24.695586 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:24.706591 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:24.707067 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:24.721218 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.126:34393
I20240502 01:52:24.721240 21346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.126:34393 every 8 connection(s)
I20240502 01:52:24.722584 21347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:24.726523 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092: Bootstrap starting.
I20240502 01:52:24.727696 21347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:24.729133 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092: No bootstrap required, opened a new log
I20240502 01:52:24.729682 21347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7652667f8dda40d8b6bdba0930be5092" member_type: VOTER }
I20240502 01:52:24.729810 21347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:24.729873 21347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7652667f8dda40d8b6bdba0930be5092, State: Initialized, Role: FOLLOWER
I20240502 01:52:24.730051 21347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7652667f8dda40d8b6bdba0930be5092" member_type: VOTER }
I20240502 01:52:24.730190 21347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:24.730247 21347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:24.730324 21347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:24.732044 21347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7652667f8dda40d8b6bdba0930be5092" member_type: VOTER }
I20240502 01:52:24.732230 21347 leader_election.cc:302] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7652667f8dda40d8b6bdba0930be5092; no voters:
I20240502 01:52:24.732579 21347 leader_election.cc:288] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:24.732710 21350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:24.733237 21350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [term 1 LEADER]: Becoming Leader. State: Replica: 7652667f8dda40d8b6bdba0930be5092, State: Running, Role: LEADER
I20240502 01:52:24.733417 21347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:24.733476 21350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7652667f8dda40d8b6bdba0930be5092" member_type: VOTER }
I20240502 01:52:24.734457 21351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7652667f8dda40d8b6bdba0930be5092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7652667f8dda40d8b6bdba0930be5092" member_type: VOTER } }
I20240502 01:52:24.734547 21352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7652667f8dda40d8b6bdba0930be5092. Latest consensus state: current_term: 1 leader_uuid: "7652667f8dda40d8b6bdba0930be5092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7652667f8dda40d8b6bdba0930be5092" member_type: VOTER } }
I20240502 01:52:24.734731 21351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:24.734758 21352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:24.735821 21355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:24.737610 21355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:24.739413 20741 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:52:24.740617 21355 catalog_manager.cc:1347] Generated new cluster ID: bccfd2ef1627430c852dded765f84d79
I20240502 01:52:24.740731 21355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:24.769232 21355 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:24.769935 21355 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:24.782056 21355 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092: Generated new TSK 0
I20240502 01:52:24.782258 21355 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:24.804004 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:24.807339 21368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:24.808037 21369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:24.808785 20741 server_base.cc:1034] running on GCE node
W20240502 01:52:24.809330 21371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:24.809645 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:24.809715 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:24.809764 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614744809738 us; error 0 us; skew 500 ppm
I20240502 01:52:24.809895 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:24.811190 20741 webserver.cc:460] Webserver started at http://127.20.65.65:33961/ using document root <none> and password file <none>
I20240502 01:52:24.811394 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:24.811470 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:24.811578 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:24.812196 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-0-root/instance:
uuid: "cee5a47493b545068259cf2cba1ce8b9"
format_stamp: "Formatted at 2024-05-02 01:52:24 on dist-test-slave-7cj2"
I20240502 01:52:24.814697 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:52:24.816219 21376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:24.816576 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:52:24.816697 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-0-root
uuid: "cee5a47493b545068259cf2cba1ce8b9"
format_stamp: "Formatted at 2024-05-02 01:52:24 on dist-test-slave-7cj2"
I20240502 01:52:24.816831 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:24.825543 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:24.826064 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:24.826637 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:24.827481 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:24.827553 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:24.827675 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:24.827737 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:24.843250 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.65:35787
I20240502 01:52:24.843273 21438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.65:35787 every 8 connection(s)
I20240502 01:52:24.844910 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:24.848434 21444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:24.848846 21445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:24.849561 21447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:24.850467 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:24.850834 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:24.850968 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:24.851034 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614744851012 us; error 0 us; skew 500 ppm
I20240502 01:52:24.851228 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:24.852443 20741 webserver.cc:460] Webserver started at http://127.20.65.66:37107/ using document root <none> and password file <none>
I20240502 01:52:24.852501 21440 heartbeater.cc:347] Connected to a master server at 127.20.65.126:34393
I20240502 01:52:24.852653 21440 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:24.852792 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:24.852936 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:24.853101 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:24.853165 21440 heartbeater.cc:511] Master 127.20.65.126:34393 requested a full tablet report, sending...
I20240502 01:52:24.853737 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-1-root/instance:
uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
format_stamp: "Formatted at 2024-05-02 01:52:24 on dist-test-slave-7cj2"
I20240502 01:52:24.854127 21312 ts_manager.cc:194] Registered new tserver with Master: cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65:35787)
I20240502 01:52:24.855790 21312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41762
I20240502 01:52:24.856941 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 01:52:24.858486 21452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:24.858840 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:52:24.858930 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-1-root
uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
format_stamp: "Formatted at 2024-05-02 01:52:24 on dist-test-slave-7cj2"
I20240502 01:52:24.859025 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:24.883668 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:24.884238 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:24.884755 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:24.885521 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:24.885603 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:24.885671 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:24.885725 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:24.900635 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.66:37353
I20240502 01:52:24.900679 21514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.66:37353 every 8 connection(s)
I20240502 01:52:24.902261 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:24.905786 21518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:24.906555 21519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:24.907272 21521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:24.907886 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:24.908205 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:24.908308 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:24.908406 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614744908364 us; error 0 us; skew 500 ppm
I20240502 01:52:24.908578 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:24.909317 21515 heartbeater.cc:347] Connected to a master server at 127.20.65.126:34393
I20240502 01:52:24.909482 21515 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:24.909842 21515 heartbeater.cc:511] Master 127.20.65.126:34393 requested a full tablet report, sending...
I20240502 01:52:24.910072 20741 webserver.cc:460] Webserver started at http://127.20.65.67:43867/ using document root <none> and password file <none>
I20240502 01:52:24.910321 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:24.910451 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:24.910599 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:24.910777 21312 ts_manager.cc:194] Registered new tserver with Master: 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353)
I20240502 01:52:24.911307 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-2-root/instance:
uuid: "d1061ad47f8c45739a726adae6735e76"
format_stamp: "Formatted at 2024-05-02 01:52:24 on dist-test-slave-7cj2"
I20240502 01:52:24.911542 21312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41772
I20240502 01:52:24.913784 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:52:24.915306 21526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:24.915628 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:52:24.915745 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-2-root
uuid: "d1061ad47f8c45739a726adae6735e76"
format_stamp: "Formatted at 2024-05-02 01:52:24 on dist-test-slave-7cj2"
I20240502 01:52:24.915875 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:24.933961 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:24.934595 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:24.935149 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:24.935935 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:24.936007 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:24.936133 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:24.936189 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:24.949512 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.67:38835
I20240502 01:52:24.949549 21590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.67:38835 every 8 connection(s)
I20240502 01:52:24.955915 21591 heartbeater.cc:347] Connected to a master server at 127.20.65.126:34393
I20240502 01:52:24.956038 21591 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:24.956403 21591 heartbeater.cc:511] Master 127.20.65.126:34393 requested a full tablet report, sending...
I20240502 01:52:24.957193 21312 ts_manager.cc:194] Registered new tserver with Master: d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:24.957933 21312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41786
I20240502 01:52:24.958221 20741 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007922164s
I20240502 01:52:25.857124 21440 heartbeater.cc:503] Master 127.20.65.126:34393 was elected leader, sending a full tablet report...
I20240502 01:52:25.912752 21515 heartbeater.cc:503] Master 127.20.65.126:34393 was elected leader, sending a full tablet report...
I20240502 01:52:25.959234 21591 heartbeater.cc:503] Master 127.20.65.126:34393 was elected leader, sending a full tablet report...
I20240502 01:52:25.986245 20741 test_util.cc:268] Using random seed: -623086454
I20240502 01:52:25.998850 21312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41798:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:52:25.999591 21312 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:52:26.036644 21480 tablet_service.cc:1467] Processing CreateTablet for tablet 1db6e77e52ff488b9631d9093ee2f78f (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 01:52:26.037420 21480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1db6e77e52ff488b9631d9093ee2f78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.037714 21404 tablet_service.cc:1467] Processing CreateTablet for tablet 1db6e77e52ff488b9631d9093ee2f78f (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 01:52:26.037744 21479 tablet_service.cc:1467] Processing CreateTablet for tablet 280305ca0fa14695b45a76e78cfff168 (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 01:52:26.037788 21478 tablet_service.cc:1467] Processing CreateTablet for tablet 225d0d79a5944db997b85fc3e4578893 (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 01:52:26.038295 21404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1db6e77e52ff488b9631d9093ee2f78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.038313 21478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 225d0d79a5944db997b85fc3e4578893. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.039093 21555 tablet_service.cc:1467] Processing CreateTablet for tablet 1db6e77e52ff488b9631d9093ee2f78f (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 01:52:26.039584 21555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1db6e77e52ff488b9631d9093ee2f78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.042686 21479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 280305ca0fa14695b45a76e78cfff168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.043002 21403 tablet_service.cc:1467] Processing CreateTablet for tablet 280305ca0fa14695b45a76e78cfff168 (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 01:52:26.043138 21402 tablet_service.cc:1467] Processing CreateTablet for tablet 225d0d79a5944db997b85fc3e4578893 (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 01:52:26.043555 21403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 280305ca0fa14695b45a76e78cfff168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.043812 21402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 225d0d79a5944db997b85fc3e4578893. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.047683 21554 tablet_service.cc:1467] Processing CreateTablet for tablet 280305ca0fa14695b45a76e78cfff168 (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 01:52:26.049064 21401 tablet_service.cc:1467] Processing CreateTablet for tablet 6c715dc7b8da49449b37addb9f9edbed (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 01:52:26.050532 21477 tablet_service.cc:1467] Processing CreateTablet for tablet 6c715dc7b8da49449b37addb9f9edbed (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 01:52:26.051807 21401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c715dc7b8da49449b37addb9f9edbed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.052386 21553 tablet_service.cc:1467] Processing CreateTablet for tablet 225d0d79a5944db997b85fc3e4578893 (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 01:52:26.053959 21554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 280305ca0fa14695b45a76e78cfff168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.054885 21400 tablet_service.cc:1467] Processing CreateTablet for tablet bd97875d904c412cbb071e82158bb046 (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 01:52:26.057485 21477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c715dc7b8da49449b37addb9f9edbed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.057046 21550 tablet_service.cc:1467] Processing CreateTablet for tablet bd97875d904c412cbb071e82158bb046 (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 01:52:26.058702 21400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd97875d904c412cbb071e82158bb046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.057382 21476 tablet_service.cc:1467] Processing CreateTablet for tablet bd97875d904c412cbb071e82158bb046 (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 01:52:26.057825 21552 tablet_service.cc:1467] Processing CreateTablet for tablet 6c715dc7b8da49449b37addb9f9edbed (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 01:52:26.061475 21474 tablet_service.cc:1467] Processing CreateTablet for tablet 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 01:52:26.063293 21475 tablet_service.cc:1467] Processing CreateTablet for tablet 3eef96a60b4145f4b5b677cfa6c91aeb (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 01:52:26.067385 21553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 225d0d79a5944db997b85fc3e4578893. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.068285 21549 tablet_service.cc:1467] Processing CreateTablet for tablet 3eef96a60b4145f4b5b677cfa6c91aeb (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 01:52:26.068727 21398 tablet_service.cc:1467] Processing CreateTablet for tablet 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 01:52:26.070895 21399 tablet_service.cc:1467] Processing CreateTablet for tablet 3eef96a60b4145f4b5b677cfa6c91aeb (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 01:52:26.071329 21550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd97875d904c412cbb071e82158bb046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.071741 21474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cca93e3d3e64bd4bb2f7dbdc9adbdb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.073937 21611 tablet_bootstrap.cc:492] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76: Bootstrap starting.
I20240502 01:52:26.074981 21476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd97875d904c412cbb071e82158bb046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.075044 21552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c715dc7b8da49449b37addb9f9edbed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.075022 21547 tablet_service.cc:1467] Processing CreateTablet for tablet 1c9c38df1bb7425d8bb9ef4fe003cd3a (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 01:52:26.077571 21548 tablet_service.cc:1467] Processing CreateTablet for tablet 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 01:52:26.078290 21549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eef96a60b4145f4b5b677cfa6c91aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.078230 21473 tablet_service.cc:1467] Processing CreateTablet for tablet 1c9c38df1bb7425d8bb9ef4fe003cd3a (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 01:52:26.077399 21475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eef96a60b4145f4b5b677cfa6c91aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.077320 21613 tablet_bootstrap.cc:492] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9: Bootstrap starting.
I20240502 01:52:26.080323 21398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cca93e3d3e64bd4bb2f7dbdc9adbdb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.081025 21548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cca93e3d3e64bd4bb2f7dbdc9adbdb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.081657 21397 tablet_service.cc:1467] Processing CreateTablet for tablet 1c9c38df1bb7425d8bb9ef4fe003cd3a (DEFAULT_TABLE table=test-workload [id=c0677fd76f604b58bcee1b9f5dbda1c2]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 01:52:26.082522 21473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c9c38df1bb7425d8bb9ef4fe003cd3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.084432 21612 tablet_bootstrap.cc:492] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc: Bootstrap starting.
I20240502 01:52:26.084534 21399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eef96a60b4145f4b5b677cfa6c91aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.087515 21397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c9c38df1bb7425d8bb9ef4fe003cd3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.087558 21547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c9c38df1bb7425d8bb9ef4fe003cd3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.105152 21611 tablet_bootstrap.cc:654] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.112579 21612 tablet_bootstrap.cc:654] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.122906 21613 tablet_bootstrap.cc:654] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.133630 21611 tablet_bootstrap.cc:492] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76: No bootstrap required, opened a new log
I20240502 01:52:26.133850 21611 ts_tablet_manager.cc:1367] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.011s
I20240502 01:52:26.134801 21611 raft_consensus.cc:348] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.135071 21611 raft_consensus.cc:374] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.135196 21611 raft_consensus.cc:729] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1061ad47f8c45739a726adae6735e76, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.135531 21611 consensus_queue.cc:260] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.136423 21611 ts_tablet_manager.cc:1398] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:52:26.136742 21611 tablet_bootstrap.cc:492] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76: Bootstrap starting.
I20240502 01:52:26.138279 21611 tablet_bootstrap.cc:654] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.140462 21613 tablet_bootstrap.cc:492] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9: No bootstrap required, opened a new log
I20240502 01:52:26.140632 21613 ts_tablet_manager.cc:1367] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.003s
I20240502 01:52:26.141427 21613 raft_consensus.cc:348] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.141677 21613 raft_consensus.cc:374] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.141793 21613 raft_consensus.cc:729] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.142061 21613 consensus_queue.cc:260] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.142872 21613 ts_tablet_manager.cc:1398] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 01:52:26.142987 21611 tablet_bootstrap.cc:492] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76: No bootstrap required, opened a new log
I20240502 01:52:26.143160 21613 tablet_bootstrap.cc:492] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9: Bootstrap starting.
I20240502 01:52:26.143162 21611 ts_tablet_manager.cc:1367] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:52:26.144013 21611 raft_consensus.cc:348] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.144289 21611 raft_consensus.cc:374] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.144573 21611 raft_consensus.cc:729] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1061ad47f8c45739a726adae6735e76, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.144805 21613 tablet_bootstrap.cc:654] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.145146 21612 tablet_bootstrap.cc:492] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc: No bootstrap required, opened a new log
I20240502 01:52:26.145298 21612 ts_tablet_manager.cc:1367] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.001s
I20240502 01:52:26.146131 21612 raft_consensus.cc:348] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.146457 21612 raft_consensus.cc:374] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.146744 21612 raft_consensus.cc:729] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e49fd5bb4f4590bcec214ff91a47cc, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.146715 21611 consensus_queue.cc:260] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.147266 21613 tablet_bootstrap.cc:492] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9: No bootstrap required, opened a new log
I20240502 01:52:26.147401 21611 ts_tablet_manager.cc:1398] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:52:26.147493 21613 ts_tablet_manager.cc:1367] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 01:52:26.147734 21611 tablet_bootstrap.cc:492] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76: Bootstrap starting.
I20240502 01:52:26.147601 21612 consensus_queue.cc:260] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.148306 21613 raft_consensus.cc:348] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.148634 21613 raft_consensus.cc:374] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.148766 21613 raft_consensus.cc:729] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.149052 21613 consensus_queue.cc:260] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.149363 21611 tablet_bootstrap.cc:654] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.149770 21613 ts_tablet_manager.cc:1398] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:52:26.150228 21613 tablet_bootstrap.cc:492] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9: Bootstrap starting.
I20240502 01:52:26.152446 21613 tablet_bootstrap.cc:654] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.154713 21612 ts_tablet_manager.cc:1398] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc: Time spent starting tablet: real 0.009s user 0.001s sys 0.007s
I20240502 01:52:26.155519 21612 tablet_bootstrap.cc:492] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc: Bootstrap starting.
I20240502 01:52:26.155916 21611 tablet_bootstrap.cc:492] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76: No bootstrap required, opened a new log
I20240502 01:52:26.156064 21611 ts_tablet_manager.cc:1367] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20240502 01:52:26.156672 21611 raft_consensus.cc:348] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.156880 21611 raft_consensus.cc:374] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.156985 21611 raft_consensus.cc:729] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1061ad47f8c45739a726adae6735e76, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.157287 21613 tablet_bootstrap.cc:492] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9: No bootstrap required, opened a new log
I20240502 01:52:26.157322 21612 tablet_bootstrap.cc:654] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.157290 21611 consensus_queue.cc:260] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.157558 21613 ts_tablet_manager.cc:1367] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240502 01:52:26.157963 21611 ts_tablet_manager.cc:1398] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:52:26.158445 21613 raft_consensus.cc:348] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.158727 21613 raft_consensus.cc:374] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.158846 21613 raft_consensus.cc:729] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.159133 21613 consensus_queue.cc:260] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.159830 21613 ts_tablet_manager.cc:1398] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:52:26.160023 21611 tablet_bootstrap.cc:492] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76: Bootstrap starting.
I20240502 01:52:26.160617 21613 tablet_bootstrap.cc:492] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9: Bootstrap starting.
I20240502 01:52:26.160821 21612 tablet_bootstrap.cc:492] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc: No bootstrap required, opened a new log
I20240502 01:52:26.160981 21612 ts_tablet_manager.cc:1367] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:52:26.161808 21612 raft_consensus.cc:348] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.162030 21612 raft_consensus.cc:374] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.162118 21611 tablet_bootstrap.cc:654] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.162122 21612 raft_consensus.cc:729] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e49fd5bb4f4590bcec214ff91a47cc, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.162349 21613 tablet_bootstrap.cc:654] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.162526 21612 consensus_queue.cc:260] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.165655 21612 ts_tablet_manager.cc:1398] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240502 01:52:26.166585 21612 tablet_bootstrap.cc:492] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc: Bootstrap starting.
I20240502 01:52:26.168709 21612 tablet_bootstrap.cc:654] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.170454 21619 raft_consensus.cc:482] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:26.170676 21619 raft_consensus.cc:504] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.171545 21611 tablet_bootstrap.cc:492] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76: No bootstrap required, opened a new log
I20240502 01:52:26.171715 21619 leader_election.cc:288] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.171944 21611 ts_tablet_manager.cc:1367] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20240502 01:52:26.172899 21613 tablet_bootstrap.cc:492] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9: No bootstrap required, opened a new log
I20240502 01:52:26.172963 21611 raft_consensus.cc:348] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.173262 21611 raft_consensus.cc:374] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.173362 21611 raft_consensus.cc:729] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1061ad47f8c45739a726adae6735e76, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.173410 21613 ts_tablet_manager.cc:1367] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20240502 01:52:26.173700 21611 consensus_queue.cc:260] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.174355 21611 ts_tablet_manager.cc:1398] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:52:26.174422 21613 raft_consensus.cc:348] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.174782 21611 tablet_bootstrap.cc:492] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76: Bootstrap starting.
I20240502 01:52:26.175277 21613 raft_consensus.cc:374] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.175458 21613 raft_consensus.cc:729] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.175820 21613 consensus_queue.cc:260] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.176980 21611 tablet_bootstrap.cc:654] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.176982 21613 ts_tablet_manager.cc:1398] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:52:26.178603 21613 tablet_bootstrap.cc:492] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9: Bootstrap starting.
I20240502 01:52:26.180763 21613 tablet_bootstrap.cc:654] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.182777 21612 tablet_bootstrap.cc:492] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc: No bootstrap required, opened a new log
I20240502 01:52:26.182924 21612 ts_tablet_manager.cc:1367] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240502 01:52:26.183670 21612 raft_consensus.cc:348] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.183873 21612 raft_consensus.cc:374] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.183985 21612 raft_consensus.cc:729] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e49fd5bb4f4590bcec214ff91a47cc, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.184341 21612 consensus_queue.cc:260] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.187156 21612 ts_tablet_manager.cc:1398] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:52:26.187537 21612 tablet_bootstrap.cc:492] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc: Bootstrap starting.
I20240502 01:52:26.187422 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db6e77e52ff488b9631d9093ee2f78f" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" is_pre_election: true
I20240502 01:52:26.187748 21490 raft_consensus.cc:2454] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 0.
I20240502 01:52:26.190618 21612 tablet_bootstrap.cc:654] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.191071 21379 leader_election.cc:302] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e49fd5bb4f4590bcec214ff91a47cc, cee5a47493b545068259cf2cba1ce8b9; no voters:
I20240502 01:52:26.191514 21619 raft_consensus.cc:2781] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:26.191653 21619 raft_consensus.cc:482] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:26.191745 21619 raft_consensus.cc:3037] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.194870 21619 raft_consensus.cc:504] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.195432 21565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db6e77e52ff488b9631d9093ee2f78f" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1061ad47f8c45739a726adae6735e76" is_pre_election: true
I20240502 01:52:26.195770 21565 raft_consensus.cc:2454] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 0.
I20240502 01:52:26.196664 21611 tablet_bootstrap.cc:492] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76: No bootstrap required, opened a new log
I20240502 01:52:26.196916 21611 ts_tablet_manager.cc:1367] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.006s
I20240502 01:52:26.197000 21565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db6e77e52ff488b9631d9093ee2f78f" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1061ad47f8c45739a726adae6735e76"
I20240502 01:52:26.197199 21613 tablet_bootstrap.cc:492] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9: No bootstrap required, opened a new log
I20240502 01:52:26.197196 21565 raft_consensus.cc:3037] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.197388 21613 ts_tablet_manager.cc:1367] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.004s
I20240502 01:52:26.198009 21611 raft_consensus.cc:348] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.198288 21611 raft_consensus.cc:374] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.198222 21613 raft_consensus.cc:348] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.198452 21613 raft_consensus.cc:374] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.198428 21611 raft_consensus.cc:729] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1061ad47f8c45739a726adae6735e76, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.198570 21613 raft_consensus.cc:729] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.198830 21613 consensus_queue.cc:260] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.198823 21611 consensus_queue.cc:260] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.200961 21565 raft_consensus.cc:2454] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 1.
I20240502 01:52:26.201550 21378 leader_election.cc:302] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cee5a47493b545068259cf2cba1ce8b9, d1061ad47f8c45739a726adae6735e76; no voters:
I20240502 01:52:26.202373 21611 ts_tablet_manager.cc:1398] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:52:26.202767 21619 leader_election.cc:288] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 election: Requested vote from peers 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.205684 21619 raft_consensus.cc:2781] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:26.206034 21611 tablet_bootstrap.cc:492] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76: Bootstrap starting.
I20240502 01:52:26.206290 21619 raft_consensus.cc:686] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Becoming Leader. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Running, Role: LEADER
I20240502 01:52:26.206965 21619 consensus_queue.cc:237] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.208403 21611 tablet_bootstrap.cc:654] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.215983 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db6e77e52ff488b9631d9093ee2f78f" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
I20240502 01:52:26.216275 21490 raft_consensus.cc:3037] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.220278 21490 raft_consensus.cc:2454] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 1.
I20240502 01:52:26.221899 21611 tablet_bootstrap.cc:492] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76: No bootstrap required, opened a new log
I20240502 01:52:26.222134 21611 ts_tablet_manager.cc:1367] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.006s
I20240502 01:52:26.223206 21611 raft_consensus.cc:348] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.223512 21611 raft_consensus.cc:374] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.223628 21611 raft_consensus.cc:729] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1061ad47f8c45739a726adae6735e76, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.224035 21611 consensus_queue.cc:260] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.226492 21310 catalog_manager.cc:5484] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65). New cstate: current_term: 1 leader_uuid: "cee5a47493b545068259cf2cba1ce8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:26.227602 21611 ts_tablet_manager.cc:1398] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240502 01:52:26.229089 21613 ts_tablet_manager.cc:1398] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9: Time spent starting tablet: real 0.032s user 0.004s sys 0.027s
I20240502 01:52:26.229413 21613 tablet_bootstrap.cc:492] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9: Bootstrap starting.
I20240502 01:52:26.230428 21612 tablet_bootstrap.cc:492] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc: No bootstrap required, opened a new log
I20240502 01:52:26.230636 21612 ts_tablet_manager.cc:1367] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.034s
I20240502 01:52:26.230999 21613 tablet_bootstrap.cc:654] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.231441 21612 raft_consensus.cc:348] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.231693 21612 raft_consensus.cc:374] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.231792 21612 raft_consensus.cc:729] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e49fd5bb4f4590bcec214ff91a47cc, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.232070 21612 consensus_queue.cc:260] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.232348 21611 tablet_bootstrap.cc:492] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76: Bootstrap starting.
I20240502 01:52:26.233376 21612 ts_tablet_manager.cc:1398] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:52:26.233736 21612 tablet_bootstrap.cc:492] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc: Bootstrap starting.
I20240502 01:52:26.233791 21616 raft_consensus.cc:482] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:26.233914 21616 raft_consensus.cc:504] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.235116 21616 leader_election.cc:288] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65:35787), 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353)
I20240502 01:52:26.237085 21611 tablet_bootstrap.cc:654] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.245842 21612 tablet_bootstrap.cc:654] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.248724 21611 tablet_bootstrap.cc:492] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76: No bootstrap required, opened a new log
I20240502 01:52:26.248899 21611 ts_tablet_manager.cc:1367] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.002s
I20240502 01:52:26.249544 21613 tablet_bootstrap.cc:492] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9: No bootstrap required, opened a new log
I20240502 01:52:26.249778 21613 ts_tablet_manager.cc:1367] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.013s
I20240502 01:52:26.249738 21611 raft_consensus.cc:348] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.250010 21611 raft_consensus.cc:374] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.250145 21611 raft_consensus.cc:729] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1061ad47f8c45739a726adae6735e76, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.250433 21611 consensus_queue.cc:260] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.251170 21611 ts_tablet_manager.cc:1398] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 01:52:26.251538 21611 tablet_bootstrap.cc:492] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76: Bootstrap starting.
I20240502 01:52:26.253217 21612 tablet_bootstrap.cc:492] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc: No bootstrap required, opened a new log
I20240502 01:52:26.253273 21611 tablet_bootstrap.cc:654] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.253381 21612 ts_tablet_manager.cc:1367] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20240502 01:52:26.254297 21612 raft_consensus.cc:348] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.254568 21612 raft_consensus.cc:374] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.254678 21612 raft_consensus.cc:729] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e49fd5bb4f4590bcec214ff91a47cc, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.254963 21612 consensus_queue.cc:260] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.255975 21613 raft_consensus.cc:348] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.256196 21613 raft_consensus.cc:374] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.256295 21613 raft_consensus.cc:729] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.256579 21613 consensus_queue.cc:260] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.257151 21613 ts_tablet_manager.cc:1398] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 01:52:26.257496 21613 tablet_bootstrap.cc:492] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9: Bootstrap starting.
I20240502 01:52:26.257710 21612 ts_tablet_manager.cc:1398] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:26.259320 21613 tablet_bootstrap.cc:654] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.264518 21612 tablet_bootstrap.cc:492] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc: Bootstrap starting.
I20240502 01:52:26.266765 21612 tablet_bootstrap.cc:654] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.267736 21613 tablet_bootstrap.cc:492] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9: No bootstrap required, opened a new log
I20240502 01:52:26.267936 21613 ts_tablet_manager.cc:1367] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20240502 01:52:26.268393 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cca93e3d3e64bd4bb2f7dbdc9adbdb8" candidate_uuid: "d1061ad47f8c45739a726adae6735e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" is_pre_election: true
I20240502 01:52:26.268916 21613 raft_consensus.cc:348] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.269183 21613 raft_consensus.cc:374] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.269299 21613 raft_consensus.cc:729] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Initialized, Role: FOLLOWER
W20240502 01:52:26.269564 21529 leader_election.cc:341] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:26.269626 21613 consensus_queue.cc:260] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.272536 21414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cca93e3d3e64bd4bb2f7dbdc9adbdb8" candidate_uuid: "d1061ad47f8c45739a726adae6735e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee5a47493b545068259cf2cba1ce8b9" is_pre_election: true
I20240502 01:52:26.272853 21414 raft_consensus.cc:2454] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1061ad47f8c45739a726adae6735e76 in term 0.
I20240502 01:52:26.272902 21613 ts_tablet_manager.cc:1398] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:52:26.273314 21613 tablet_bootstrap.cc:492] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9: Bootstrap starting.
I20240502 01:52:26.273438 21527 leader_election.cc:302] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cee5a47493b545068259cf2cba1ce8b9, d1061ad47f8c45739a726adae6735e76; no voters: 54e49fd5bb4f4590bcec214ff91a47cc
I20240502 01:52:26.273941 21616 raft_consensus.cc:2781] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:26.274060 21616 raft_consensus.cc:482] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:26.274163 21616 raft_consensus.cc:3037] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.275656 21613 tablet_bootstrap.cc:654] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.276639 21616 raft_consensus.cc:504] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.277827 21616 leader_election.cc:288] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [CANDIDATE]: Term 1 election: Requested vote from peers cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65:35787), 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353)
I20240502 01:52:26.277796 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cca93e3d3e64bd4bb2f7dbdc9adbdb8" candidate_uuid: "d1061ad47f8c45739a726adae6735e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
I20240502 01:52:26.277936 21414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cca93e3d3e64bd4bb2f7dbdc9adbdb8" candidate_uuid: "d1061ad47f8c45739a726adae6735e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee5a47493b545068259cf2cba1ce8b9"
I20240502 01:52:26.278115 21414 raft_consensus.cc:3037] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.278149 21612 tablet_bootstrap.cc:492] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc: No bootstrap required, opened a new log
I20240502 01:52:26.278327 21612 ts_tablet_manager.cc:1367] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
W20240502 01:52:26.278501 21529 leader_election.cc:341] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:26.279011 21611 tablet_bootstrap.cc:492] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76: No bootstrap required, opened a new log
I20240502 01:52:26.279037 21612 raft_consensus.cc:348] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.279196 21611 ts_tablet_manager.cc:1367] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.016s
I20240502 01:52:26.279264 21612 raft_consensus.cc:374] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.279376 21612 raft_consensus.cc:729] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e49fd5bb4f4590bcec214ff91a47cc, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.279686 21612 consensus_queue.cc:260] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.280025 21611 raft_consensus.cc:348] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.280272 21611 raft_consensus.cc:374] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.280373 21611 raft_consensus.cc:729] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1061ad47f8c45739a726adae6735e76, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.280640 21611 consensus_queue.cc:260] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.280907 21613 tablet_bootstrap.cc:492] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9: No bootstrap required, opened a new log
I20240502 01:52:26.281129 21613 ts_tablet_manager.cc:1367] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240502 01:52:26.281411 21414 raft_consensus.cc:2454] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1061ad47f8c45739a726adae6735e76 in term 1.
I20240502 01:52:26.282037 21527 leader_election.cc:302] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cee5a47493b545068259cf2cba1ce8b9, d1061ad47f8c45739a726adae6735e76; no voters: 54e49fd5bb4f4590bcec214ff91a47cc
I20240502 01:52:26.282313 21611 ts_tablet_manager.cc:1398] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:52:26.282239 21613 raft_consensus.cc:348] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.282522 21616 raft_consensus.cc:2781] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:26.282639 21613 raft_consensus.cc:374] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.282786 21613 raft_consensus.cc:729] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.283084 21613 consensus_queue.cc:260] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.283756 21613 ts_tablet_manager.cc:1398] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:52:26.284016 21612 ts_tablet_manager.cc:1398] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240502 01:52:26.286571 21633 raft_consensus.cc:482] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:26.286747 21633 raft_consensus.cc:504] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.287786 21633 leader_election.cc:288] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.288178 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225d0d79a5944db997b85fc3e4578893" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" is_pre_election: true
I20240502 01:52:26.288393 21565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225d0d79a5944db997b85fc3e4578893" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1061ad47f8c45739a726adae6735e76" is_pre_election: true
I20240502 01:52:26.288489 21490 raft_consensus.cc:2454] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 0.
I20240502 01:52:26.288628 21565 raft_consensus.cc:2454] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 0.
I20240502 01:52:26.288846 21612 tablet_bootstrap.cc:492] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc: Bootstrap starting.
I20240502 01:52:26.289078 21378 leader_election.cc:302] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cee5a47493b545068259cf2cba1ce8b9, d1061ad47f8c45739a726adae6735e76; no voters:
I20240502 01:52:26.289651 21633 raft_consensus.cc:2781] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:26.289788 21633 raft_consensus.cc:482] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:26.289878 21633 raft_consensus.cc:3037] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.290719 21612 tablet_bootstrap.cc:654] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.293506 21633 raft_consensus.cc:504] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.294219 21616 raft_consensus.cc:686] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [term 1 LEADER]: Becoming Leader. State: Replica: d1061ad47f8c45739a726adae6735e76, State: Running, Role: LEADER
I20240502 01:52:26.294538 21616 consensus_queue.cc:237] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.294763 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225d0d79a5944db997b85fc3e4578893" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
I20240502 01:52:26.294950 21490 raft_consensus.cc:3037] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.295270 21565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225d0d79a5944db997b85fc3e4578893" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1061ad47f8c45739a726adae6735e76"
I20240502 01:52:26.295543 21565 raft_consensus.cc:3037] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.297564 21619 raft_consensus.cc:482] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:26.297775 21619 raft_consensus.cc:504] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.299062 21489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c715dc7b8da49449b37addb9f9edbed" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" is_pre_election: true
I20240502 01:52:26.299424 21489 raft_consensus.cc:2454] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 0.
I20240502 01:52:26.299590 21565 raft_consensus.cc:2454] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 1.
I20240502 01:52:26.299912 21379 leader_election.cc:302] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e49fd5bb4f4590bcec214ff91a47cc, cee5a47493b545068259cf2cba1ce8b9; no voters:
I20240502 01:52:26.300279 21634 raft_consensus.cc:2781] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:26.300357 21378 leader_election.cc:302] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cee5a47493b545068259cf2cba1ce8b9, d1061ad47f8c45739a726adae6735e76; no voters:
I20240502 01:52:26.300419 21634 raft_consensus.cc:482] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:26.300854 21634 raft_consensus.cc:3037] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.301676 21621 raft_consensus.cc:482] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:26.301803 21621 raft_consensus.cc:504] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.302145 21490 raft_consensus.cc:2454] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 1.
I20240502 01:52:26.302947 21621 leader_election.cc:288] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65:35787), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.304663 21634 raft_consensus.cc:504] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.304873 21612 tablet_bootstrap.cc:492] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc: No bootstrap required, opened a new log
I20240502 01:52:26.305019 21612 ts_tablet_manager.cc:1367] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.011s
I20240502 01:52:26.305677 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c715dc7b8da49449b37addb9f9edbed" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
I20240502 01:52:26.305856 21490 raft_consensus.cc:3037] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.305881 21612 raft_consensus.cc:348] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.306053 21612 raft_consensus.cc:374] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.306157 21612 raft_consensus.cc:729] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e49fd5bb4f4590bcec214ff91a47cc, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.306377 21612 consensus_queue.cc:260] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.306955 21612 ts_tablet_manager.cc:1398] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:26.308058 21612 tablet_bootstrap.cc:492] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc: Bootstrap starting.
I20240502 01:52:26.308794 21490 raft_consensus.cc:2454] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 1.
I20240502 01:52:26.309232 21379 leader_election.cc:302] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e49fd5bb4f4590bcec214ff91a47cc, cee5a47493b545068259cf2cba1ce8b9; no voters:
I20240502 01:52:26.300691 21619 leader_election.cc:288] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.310221 21619 raft_consensus.cc:2781] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:26.310341 21612 tablet_bootstrap.cc:654] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.310367 21619 raft_consensus.cc:686] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Becoming Leader. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Running, Role: LEADER
I20240502 01:52:26.310698 21619 consensus_queue.cc:237] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.318114 21653 raft_consensus.cc:2781] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:26.319061 21634 leader_election.cc:288] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 election: Requested vote from peers 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.319375 21633 leader_election.cc:288] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 election: Requested vote from peers 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.320748 21310 catalog_manager.cc:5484] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1061ad47f8c45739a726adae6735e76 (127.20.65.67). New cstate: current_term: 1 leader_uuid: "d1061ad47f8c45739a726adae6735e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:26.327008 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c715dc7b8da49449b37addb9f9edbed" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1061ad47f8c45739a726adae6735e76"
I20240502 01:52:26.327322 21564 raft_consensus.cc:3037] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.328264 21565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c715dc7b8da49449b37addb9f9edbed" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1061ad47f8c45739a726adae6735e76" is_pre_election: true
I20240502 01:52:26.329826 21653 raft_consensus.cc:686] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Becoming Leader. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Running, Role: LEADER
I20240502 01:52:26.331104 21616 raft_consensus.cc:482] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:26.331521 21565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280305ca0fa14695b45a76e78cfff168" candidate_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1061ad47f8c45739a726adae6735e76" is_pre_election: true
I20240502 01:52:26.331476 21616 raft_consensus.cc:504] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.332032 21612 tablet_bootstrap.cc:492] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc: No bootstrap required, opened a new log
I20240502 01:52:26.332289 21616 leader_election.cc:288] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65:35787), 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353)
I20240502 01:52:26.333045 21414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280305ca0fa14695b45a76e78cfff168" candidate_uuid: "d1061ad47f8c45739a726adae6735e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee5a47493b545068259cf2cba1ce8b9" is_pre_election: true
I20240502 01:52:26.333180 21565 raft_consensus.cc:2454] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54e49fd5bb4f4590bcec214ff91a47cc in term 0.
I20240502 01:52:26.333137 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280305ca0fa14695b45a76e78cfff168" candidate_uuid: "d1061ad47f8c45739a726adae6735e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" is_pre_election: true
I20240502 01:52:26.333295 21414 raft_consensus.cc:2454] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1061ad47f8c45739a726adae6735e76 in term 0.
I20240502 01:52:26.333382 21490 raft_consensus.cc:2454] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1061ad47f8c45739a726adae6735e76 in term 0.
I20240502 01:52:26.333410 21653 consensus_queue.cc:237] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.333738 21454 leader_election.cc:302] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e49fd5bb4f4590bcec214ff91a47cc, d1061ad47f8c45739a726adae6735e76; no voters:
I20240502 01:52:26.334059 21621 raft_consensus.cc:2781] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:26.334070 21564 raft_consensus.cc:2454] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 1.
I20240502 01:52:26.334264 21527 leader_election.cc:302] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cee5a47493b545068259cf2cba1ce8b9, d1061ad47f8c45739a726adae6735e76; no voters:
I20240502 01:52:26.334564 21621 raft_consensus.cc:482] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:26.334641 21612 ts_tablet_manager.cc:1367] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.012s
I20240502 01:52:26.334708 21621 raft_consensus.cc:3037] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.334196 21413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280305ca0fa14695b45a76e78cfff168" candidate_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee5a47493b545068259cf2cba1ce8b9" is_pre_election: true
I20240502 01:52:26.335122 21413 raft_consensus.cc:2454] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54e49fd5bb4f4590bcec214ff91a47cc in term 0.
I20240502 01:52:26.335597 21612 raft_consensus.cc:348] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.335837 21612 raft_consensus.cc:374] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.335929 21612 raft_consensus.cc:729] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e49fd5bb4f4590bcec214ff91a47cc, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.336169 21612 consensus_queue.cc:260] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.337066 21612 ts_tablet_manager.cc:1398] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:52:26.337742 21621 raft_consensus.cc:504] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.338824 21312 catalog_manager.cc:5484] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65). New cstate: current_term: 1 leader_uuid: "cee5a47493b545068259cf2cba1ce8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:26.339246 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280305ca0fa14695b45a76e78cfff168" candidate_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1061ad47f8c45739a726adae6735e76"
I20240502 01:52:26.339356 21616 raft_consensus.cc:2781] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:26.339500 21564 raft_consensus.cc:3037] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.339601 21621 leader_election.cc:288] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [CANDIDATE]: Term 1 election: Requested vote from peers cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65:35787), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.339457 21312 catalog_manager.cc:5484] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65). New cstate: current_term: 1 leader_uuid: "cee5a47493b545068259cf2cba1ce8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:26.341820 21564 raft_consensus.cc:2454] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e49fd5bb4f4590bcec214ff91a47cc in term 1.
I20240502 01:52:26.341995 21616 raft_consensus.cc:482] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:26.342186 21616 raft_consensus.cc:3037] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:26.342298 21454 leader_election.cc:302] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e49fd5bb4f4590bcec214ff91a47cc, d1061ad47f8c45739a726adae6735e76; no voters:
I20240502 01:52:26.342196 21413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280305ca0fa14695b45a76e78cfff168" candidate_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee5a47493b545068259cf2cba1ce8b9"
I20240502 01:52:26.342602 21413 raft_consensus.cc:3037] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.342736 21621 raft_consensus.cc:2781] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:26.342936 21621 raft_consensus.cc:686] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 LEADER]: Becoming Leader. State: Replica: 54e49fd5bb4f4590bcec214ff91a47cc, State: Running, Role: LEADER
I20240502 01:52:26.343322 21621 consensus_queue.cc:237] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.345085 21616 raft_consensus.cc:504] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.345288 21413 raft_consensus.cc:2454] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e49fd5bb4f4590bcec214ff91a47cc in term 1.
I20240502 01:52:26.346396 21616 leader_election.cc:288] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [CANDIDATE]: Term 2 election: Requested vote from peers cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65:35787), 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353)
I20240502 01:52:26.346683 21413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280305ca0fa14695b45a76e78cfff168" candidate_uuid: "d1061ad47f8c45739a726adae6735e76" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee5a47493b545068259cf2cba1ce8b9"
I20240502 01:52:26.346866 21413 raft_consensus.cc:3037] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:26.346839 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280305ca0fa14695b45a76e78cfff168" candidate_uuid: "d1061ad47f8c45739a726adae6735e76" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
I20240502 01:52:26.348939 21413 raft_consensus.cc:2454] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1061ad47f8c45739a726adae6735e76 in term 2.
I20240502 01:52:26.348803 21312 catalog_manager.cc:5484] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66). New cstate: current_term: 1 leader_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:26.349339 21527 leader_election.cc:302] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cee5a47493b545068259cf2cba1ce8b9, d1061ad47f8c45739a726adae6735e76; no voters: 54e49fd5bb4f4590bcec214ff91a47cc
I20240502 01:52:26.349442 21653 raft_consensus.cc:482] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:26.349617 21653 raft_consensus.cc:504] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.349730 21616 raft_consensus.cc:2781] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:26.349895 21616 raft_consensus.cc:686] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 2 LEADER]: Becoming Leader. State: Replica: d1061ad47f8c45739a726adae6735e76, State: Running, Role: LEADER
I20240502 01:52:26.350158 21616 consensus_queue.cc:237] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.350272 21653 leader_election.cc:288] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.350666 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c9c38df1bb7425d8bb9ef4fe003cd3a" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" is_pre_election: true
I20240502 01:52:26.352022 21490 raft_consensus.cc:2454] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 0.
I20240502 01:52:26.351159 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c9c38df1bb7425d8bb9ef4fe003cd3a" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1061ad47f8c45739a726adae6735e76" is_pre_election: true
I20240502 01:52:26.352450 21564 raft_consensus.cc:2454] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 0.
I20240502 01:52:26.352488 21379 leader_election.cc:302] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e49fd5bb4f4590bcec214ff91a47cc, cee5a47493b545068259cf2cba1ce8b9; no voters:
I20240502 01:52:26.352785 21653 raft_consensus.cc:2781] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:26.352888 21653 raft_consensus.cc:482] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:26.353003 21653 raft_consensus.cc:3037] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.353401 21312 catalog_manager.cc:5484] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 reported cstate change: term changed from 1 to 2, leader changed from 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66) to d1061ad47f8c45739a726adae6735e76 (127.20.65.67). New cstate: current_term: 2 leader_uuid: "d1061ad47f8c45739a726adae6735e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:26.355530 21653 raft_consensus.cc:504] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.356087 21653 leader_election.cc:288] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 election: Requested vote from peers 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.356449 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c9c38df1bb7425d8bb9ef4fe003cd3a" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
I20240502 01:52:26.356552 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c9c38df1bb7425d8bb9ef4fe003cd3a" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1061ad47f8c45739a726adae6735e76"
I20240502 01:52:26.356642 21490 raft_consensus.cc:3037] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.356748 21564 raft_consensus.cc:3037] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.358672 21490 raft_consensus.cc:2454] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 1.
I20240502 01:52:26.358744 21564 raft_consensus.cc:2454] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 1.
I20240502 01:52:26.359149 21378 leader_election.cc:302] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cee5a47493b545068259cf2cba1ce8b9, d1061ad47f8c45739a726adae6735e76; no voters:
I20240502 01:52:26.359402 21653 raft_consensus.cc:2781] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:26.359596 21653 raft_consensus.cc:686] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Becoming Leader. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Running, Role: LEADER
I20240502 01:52:26.359902 21653 consensus_queue.cc:237] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.362123 21312 catalog_manager.cc:5484] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65). New cstate: current_term: 1 leader_uuid: "cee5a47493b545068259cf2cba1ce8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:26.382944 21653 raft_consensus.cc:482] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:26.383145 21653 raft_consensus.cc:504] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.383891 21653 leader_election.cc:288] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.384409 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eef96a60b4145f4b5b677cfa6c91aeb" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1061ad47f8c45739a726adae6735e76" is_pre_election: true
I20240502 01:52:26.384409 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eef96a60b4145f4b5b677cfa6c91aeb" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" is_pre_election: true
I20240502 01:52:26.384708 21564 raft_consensus.cc:2454] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 0.
I20240502 01:52:26.384714 21490 raft_consensus.cc:2454] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 0.
I20240502 01:52:26.385113 21378 leader_election.cc:302] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cee5a47493b545068259cf2cba1ce8b9, d1061ad47f8c45739a726adae6735e76; no voters:
I20240502 01:52:26.385454 21653 raft_consensus.cc:2781] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:26.385553 21653 raft_consensus.cc:482] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:26.385645 21653 raft_consensus.cc:3037] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.386482 21621 raft_consensus.cc:482] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:26.386670 21621 raft_consensus.cc:504] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.387465 21621 leader_election.cc:288] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65:35787), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.387773 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cca93e3d3e64bd4bb2f7dbdc9adbdb8" candidate_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1061ad47f8c45739a726adae6735e76" is_pre_election: true
I20240502 01:52:26.387812 21413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cca93e3d3e64bd4bb2f7dbdc9adbdb8" candidate_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee5a47493b545068259cf2cba1ce8b9" is_pre_election: true
I20240502 01:52:26.388131 21413 raft_consensus.cc:2379] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54e49fd5bb4f4590bcec214ff91a47cc in current term 1: Already voted for candidate d1061ad47f8c45739a726adae6735e76 in this term.
I20240502 01:52:26.388434 21653 raft_consensus.cc:504] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.388617 21453 leader_election.cc:302] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54e49fd5bb4f4590bcec214ff91a47cc; no voters: cee5a47493b545068259cf2cba1ce8b9, d1061ad47f8c45739a726adae6735e76
I20240502 01:52:26.388919 21621 raft_consensus.cc:3037] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.389219 21653 leader_election.cc:288] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 election: Requested vote from peers 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.389488 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eef96a60b4145f4b5b677cfa6c91aeb" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
I20240502 01:52:26.389535 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eef96a60b4145f4b5b677cfa6c91aeb" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1061ad47f8c45739a726adae6735e76"
I20240502 01:52:26.389731 21564 raft_consensus.cc:3037] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.389736 21490 raft_consensus.cc:3037] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.391849 21564 raft_consensus.cc:2454] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 1.
I20240502 01:52:26.392184 21378 leader_election.cc:302] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cee5a47493b545068259cf2cba1ce8b9, d1061ad47f8c45739a726adae6735e76; no voters:
I20240502 01:52:26.392194 21621 raft_consensus.cc:2726] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:52:26.392458 21653 raft_consensus.cc:2781] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:26.392587 21653 raft_consensus.cc:686] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Becoming Leader. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Running, Role: LEADER
I20240502 01:52:26.392846 21490 raft_consensus.cc:2454] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 1.
I20240502 01:52:26.392864 21653 consensus_queue.cc:237] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.395539 21312 catalog_manager.cc:5484] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65). New cstate: current_term: 1 leader_uuid: "cee5a47493b545068259cf2cba1ce8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:26.516947 21653 raft_consensus.cc:482] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:26.517199 21653 raft_consensus.cc:504] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.517894 21653 leader_election.cc:288] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.518515 21616 raft_consensus.cc:482] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:26.518518 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd97875d904c412cbb071e82158bb046" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" is_pre_election: true
I20240502 01:52:26.518500 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd97875d904c412cbb071e82158bb046" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1061ad47f8c45739a726adae6735e76" is_pre_election: true
I20240502 01:52:26.518808 21490 raft_consensus.cc:2454] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 0.
I20240502 01:52:26.518752 21616 raft_consensus.cc:504] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.519137 21564 raft_consensus.cc:2454] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 0.
I20240502 01:52:26.519472 21616 leader_election.cc:288] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65:35787), 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353)
I20240502 01:52:26.519481 21379 leader_election.cc:302] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e49fd5bb4f4590bcec214ff91a47cc, cee5a47493b545068259cf2cba1ce8b9; no voters:
I20240502 01:52:26.519837 21413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd97875d904c412cbb071e82158bb046" candidate_uuid: "d1061ad47f8c45739a726adae6735e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee5a47493b545068259cf2cba1ce8b9" is_pre_election: true
I20240502 01:52:26.519929 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd97875d904c412cbb071e82158bb046" candidate_uuid: "d1061ad47f8c45739a726adae6735e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" is_pre_election: true
I20240502 01:52:26.520036 21413 raft_consensus.cc:2454] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1061ad47f8c45739a726adae6735e76 in term 0.
I20240502 01:52:26.520069 21490 raft_consensus.cc:2454] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1061ad47f8c45739a726adae6735e76 in term 0.
I20240502 01:52:26.520351 21653 raft_consensus.cc:2781] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:26.520452 21653 raft_consensus.cc:482] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:26.520464 21529 leader_election.cc:302] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e49fd5bb4f4590bcec214ff91a47cc, d1061ad47f8c45739a726adae6735e76; no voters:
I20240502 01:52:26.520572 21653 raft_consensus.cc:3037] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.520763 21616 raft_consensus.cc:2781] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:26.520891 21616 raft_consensus.cc:482] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:26.520987 21616 raft_consensus.cc:3037] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.522837 21653 raft_consensus.cc:504] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.522938 21616 raft_consensus.cc:504] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.523373 21653 leader_election.cc:288] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 election: Requested vote from peers 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.523550 21616 leader_election.cc:288] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [CANDIDATE]: Term 1 election: Requested vote from peers cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65:35787), 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66:37353)
I20240502 01:52:26.523797 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd97875d904c412cbb071e82158bb046" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
I20240502 01:52:26.523814 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd97875d904c412cbb071e82158bb046" candidate_uuid: "cee5a47493b545068259cf2cba1ce8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1061ad47f8c45739a726adae6735e76"
I20240502 01:52:26.523892 21413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd97875d904c412cbb071e82158bb046" candidate_uuid: "d1061ad47f8c45739a726adae6735e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee5a47493b545068259cf2cba1ce8b9"
I20240502 01:52:26.524010 21490 raft_consensus.cc:3037] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.524086 21564 raft_consensus.cc:2379] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cee5a47493b545068259cf2cba1ce8b9 in current term 1: Already voted for candidate d1061ad47f8c45739a726adae6735e76 in this term.
I20240502 01:52:26.524137 21413 raft_consensus.cc:2379] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d1061ad47f8c45739a726adae6735e76 in current term 1: Already voted for candidate cee5a47493b545068259cf2cba1ce8b9 in this term.
I20240502 01:52:26.525866 21490 raft_consensus.cc:2454] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee5a47493b545068259cf2cba1ce8b9 in term 1.
I20240502 01:52:26.526161 21379 leader_election.cc:302] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54e49fd5bb4f4590bcec214ff91a47cc, cee5a47493b545068259cf2cba1ce8b9; no voters: d1061ad47f8c45739a726adae6735e76
I20240502 01:52:26.526393 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd97875d904c412cbb071e82158bb046" candidate_uuid: "d1061ad47f8c45739a726adae6735e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
I20240502 01:52:26.526454 21653 raft_consensus.cc:2781] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:26.526685 21490 raft_consensus.cc:2379] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d1061ad47f8c45739a726adae6735e76 in current term 1: Already voted for candidate cee5a47493b545068259cf2cba1ce8b9 in this term.
I20240502 01:52:26.526657 21653 raft_consensus.cc:686] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Becoming Leader. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Running, Role: LEADER
I20240502 01:52:26.527011 21653 consensus_queue.cc:237] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.527060 21529 leader_election.cc:302] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1061ad47f8c45739a726adae6735e76; no voters: 54e49fd5bb4f4590bcec214ff91a47cc, cee5a47493b545068259cf2cba1ce8b9
I20240502 01:52:26.527324 21616 raft_consensus.cc:2726] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:52:26.529537 21312 catalog_manager.cc:5484] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65). New cstate: current_term: 1 leader_uuid: "cee5a47493b545068259cf2cba1ce8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:26.629825 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:26.633106 21664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:26.633725 21665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:26.634428 21667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:26.634999 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:26.635298 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:26.635380 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:26.635422 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614746635409 us; error 0 us; skew 500 ppm
I20240502 01:52:26.635566 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:26.641283 20741 webserver.cc:460] Webserver started at http://127.20.65.68:35101/ using document root <none> and password file <none>
I20240502 01:52:26.641513 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:26.641584 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:26.641711 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:26.642346 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-3-root/instance:
uuid: "58b53c132697421181f99cda363311ef"
format_stamp: "Formatted at 2024-05-02 01:52:26 on dist-test-slave-7cj2"
I20240502 01:52:26.644945 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:26.646576 21672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:26.646991 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 01:52:26.647099 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-3-root
uuid: "58b53c132697421181f99cda363311ef"
format_stamp: "Formatted at 2024-05-02 01:52:26 on dist-test-slave-7cj2"
I20240502 01:52:26.647199 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614741486931-20741-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:26.660095 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:26.660672 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:26.661123 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:26.661913 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:26.661989 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:26.662056 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:26.662108 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:26.678978 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.68:35677
I20240502 01:52:26.679035 21735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.68:35677 every 8 connection(s)
I20240502 01:52:26.683508 21616 consensus_queue.cc:1035] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:26.686827 21736 heartbeater.cc:347] Connected to a master server at 127.20.65.126:34393
I20240502 01:52:26.686968 21736 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:26.687476 21736 heartbeater.cc:511] Master 127.20.65.126:34393 requested a full tablet report, sending...
I20240502 01:52:26.688481 21312 ts_manager.cc:194] Registered new tserver with Master: 58b53c132697421181f99cda363311ef (127.20.65.68:35677)
I20240502 01:52:26.689834 21312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41802
I20240502 01:52:26.694510 21616 consensus_queue.cc:1035] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:26.735345 21564 raft_consensus.cc:1226] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 2 LEADER]: Rejecting Update request from peer 54e49fd5bb4f4590bcec214ff91a47cc for earlier term 1. Current term is 2. Ops: []
I20240502 01:52:26.735957 21621 consensus_queue.cc:1046] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:26.736233 21621 consensus_queue.cc:1225] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [LEADER]: Peer d1061ad47f8c45739a726adae6735e76 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240502 01:52:26.736413 21658 raft_consensus.cc:3032] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:52:26.736511 21658 raft_consensus.cc:729] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 54e49fd5bb4f4590bcec214ff91a47cc, State: Running, Role: LEADER
I20240502 01:52:26.736862 21658 consensus_queue.cc:260] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.737154 21658 raft_consensus.cc:3037] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:26.739351 21365 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:52:26.745903 21653 consensus_queue.cc:1035] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:26.748070 21413 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1db6e77e52ff488b9631d9093ee2f78f"
dest_uuid: "cee5a47493b545068259cf2cba1ce8b9"
mode: GRACEFUL
new_leader_uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
from {username='slave'} at 127.0.0.1:59980
I20240502 01:52:26.748395 21413 raft_consensus.cc:595] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Received request to transfer leadership to TS 54e49fd5bb4f4590bcec214ff91a47cc
I20240502 01:52:26.750644 21656 consensus_queue.cc:1035] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:26.751825 21413 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "225d0d79a5944db997b85fc3e4578893"
dest_uuid: "cee5a47493b545068259cf2cba1ce8b9"
mode: GRACEFUL
new_leader_uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
from {username='slave'} at 127.0.0.1:59980
I20240502 01:52:26.752050 21413 raft_consensus.cc:595] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Received request to transfer leadership to TS 54e49fd5bb4f4590bcec214ff91a47cc
I20240502 01:52:26.753677 21656 consensus_queue.cc:1035] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:26.754109 21413 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6c715dc7b8da49449b37addb9f9edbed"
dest_uuid: "cee5a47493b545068259cf2cba1ce8b9"
mode: GRACEFUL
new_leader_uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
from {username='slave'} at 127.0.0.1:59980
I20240502 01:52:26.754303 21413 raft_consensus.cc:595] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Received request to transfer leadership to TS 54e49fd5bb4f4590bcec214ff91a47cc
I20240502 01:52:26.754359 21650 raft_consensus.cc:979] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9: : Instructing follower 54e49fd5bb4f4590bcec214ff91a47cc to start an election
I20240502 01:52:26.754542 21650 raft_consensus.cc:1067] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Signalling peer 54e49fd5bb4f4590bcec214ff91a47cc to start an election
I20240502 01:52:26.754770 21365 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 01:52:26.755123 21365 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:52:26.755585 21489 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "225d0d79a5944db997b85fc3e4578893"
dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
from {username='slave'} at 127.0.0.1:47744
I20240502 01:52:26.755821 21489 raft_consensus.cc:482] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:52:26.755959 21489 raft_consensus.cc:3037] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:26.756829 21650 raft_consensus.cc:979] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9: : Instructing follower 54e49fd5bb4f4590bcec214ff91a47cc to start an election
I20240502 01:52:26.756976 21650 raft_consensus.cc:1067] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Signalling peer 54e49fd5bb4f4590bcec214ff91a47cc to start an election
I20240502 01:52:26.757814 21488 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1db6e77e52ff488b9631d9093ee2f78f"
dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
from {username='slave'} at 127.0.0.1:47744
I20240502 01:52:26.758050 21488 raft_consensus.cc:482] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:52:26.758181 21488 raft_consensus.cc:3037] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:26.758915 21489 raft_consensus.cc:504] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.759964 21413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225d0d79a5944db997b85fc3e4578893" candidate_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cee5a47493b545068259cf2cba1ce8b9"
I20240502 01:52:26.760200 21413 raft_consensus.cc:3032] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:52:26.760375 21413 raft_consensus.cc:729] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Running, Role: LEADER
I20240502 01:52:26.760572 21488 raft_consensus.cc:504] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.760835 21413 consensus_queue.cc:260] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.761289 21413 raft_consensus.cc:3037] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:26.761379 21650 consensus_queue.cc:1035] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:26.761929 21414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db6e77e52ff488b9631d9093ee2f78f" candidate_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cee5a47493b545068259cf2cba1ce8b9"
I20240502 01:52:26.762163 21488 leader_election.cc:288] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [CANDIDATE]: Term 2 election: Requested vote from peers cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65:35787), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.762298 21414 raft_consensus.cc:3032] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:52:26.762424 21414 raft_consensus.cc:729] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Running, Role: LEADER
I20240502 01:52:26.762676 21414 consensus_queue.cc:260] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.763058 21414 raft_consensus.cc:3037] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:26.764609 21413 raft_consensus.cc:2454] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e49fd5bb4f4590bcec214ff91a47cc in term 2.
I20240502 01:52:26.765192 21453 leader_election.cc:302] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e49fd5bb4f4590bcec214ff91a47cc, cee5a47493b545068259cf2cba1ce8b9; no voters:
I20240502 01:52:26.765544 21658 raft_consensus.cc:2781] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:26.765708 21658 raft_consensus.cc:686] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 LEADER]: Becoming Leader. State: Replica: 54e49fd5bb4f4590bcec214ff91a47cc, State: Running, Role: LEADER
I20240502 01:52:26.765853 21414 raft_consensus.cc:2454] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e49fd5bb4f4590bcec214ff91a47cc in term 2.
I20240502 01:52:26.766034 21658 consensus_queue.cc:237] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.766227 21453 leader_election.cc:302] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e49fd5bb4f4590bcec214ff91a47cc, cee5a47493b545068259cf2cba1ce8b9; no voters:
I20240502 01:52:26.766767 21563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db6e77e52ff488b9631d9093ee2f78f" candidate_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1061ad47f8c45739a726adae6735e76"
I20240502 01:52:26.766983 21658 raft_consensus.cc:2781] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:26.767127 21658 raft_consensus.cc:686] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 LEADER]: Becoming Leader. State: Replica: 54e49fd5bb4f4590bcec214ff91a47cc, State: Running, Role: LEADER
I20240502 01:52:26.767405 21658 consensus_queue.cc:237] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.768899 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225d0d79a5944db997b85fc3e4578893" candidate_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1061ad47f8c45739a726adae6735e76"
I20240502 01:52:26.769086 21564 raft_consensus.cc:3037] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:26.770579 21312 catalog_manager.cc:5484] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc reported cstate change: term changed from 1 to 2, leader changed from cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65) to 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66). New cstate: current_term: 2 leader_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:26.771034 21312 catalog_manager.cc:5484] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc reported cstate change: term changed from 1 to 2, leader changed from cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65) to 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66). New cstate: current_term: 2 leader_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:26.771188 21564 raft_consensus.cc:2454] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e49fd5bb4f4590bcec214ff91a47cc in term 2.
I20240502 01:52:26.772357 21489 leader_election.cc:288] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [CANDIDATE]: Term 2 election: Requested vote from peers cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65:35787), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.818795 21414 raft_consensus.cc:1261] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9 [term 2 FOLLOWER]: Refusing update from remote peer d1061ad47f8c45739a726adae6735e76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 01:52:26.819402 21616 consensus_queue.cc:1035] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:26.823391 21489 raft_consensus.cc:1261] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 FOLLOWER]: Refusing update from remote peer d1061ad47f8c45739a726adae6735e76: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20240502 01:52:26.823549 21489 pending_rounds.cc:76] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc: Aborting all ops after (but not including) 0
I20240502 01:52:26.823657 21489 pending_rounds.cc:97] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240502 01:52:26.823827 21489 raft_consensus.cc:2866] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:52:26.824394 21751 consensus_queue.cc:1035] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:26.827140 21653 consensus_queue.cc:1035] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:26.827383 21650 consensus_queue.cc:1035] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:26.836566 21653 consensus_queue.cc:1035] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:26.840618 21656 consensus_queue.cc:1035] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:26.843061 21656 raft_consensus.cc:979] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9: : Instructing follower 54e49fd5bb4f4590bcec214ff91a47cc to start an election
I20240502 01:52:26.843242 21656 raft_consensus.cc:1067] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Signalling peer 54e49fd5bb4f4590bcec214ff91a47cc to start an election
I20240502 01:52:26.843822 21489 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c715dc7b8da49449b37addb9f9edbed"
dest_uuid: "54e49fd5bb4f4590bcec214ff91a47cc"
from {username='slave'} at 127.0.0.1:47744
I20240502 01:52:26.844036 21489 raft_consensus.cc:482] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:52:26.844161 21489 raft_consensus.cc:3037] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:26.846140 21489 raft_consensus.cc:504] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.846768 21489 leader_election.cc:288] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [CANDIDATE]: Term 2 election: Requested vote from peers cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65:35787), d1061ad47f8c45739a726adae6735e76 (127.20.65.67:38835)
I20240502 01:52:26.847138 21414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c715dc7b8da49449b37addb9f9edbed" candidate_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cee5a47493b545068259cf2cba1ce8b9"
I20240502 01:52:26.847335 21414 raft_consensus.cc:3032] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:52:26.847455 21414 raft_consensus.cc:729] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cee5a47493b545068259cf2cba1ce8b9, State: Running, Role: LEADER
I20240502 01:52:26.847721 21414 consensus_queue.cc:260] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.848125 21414 raft_consensus.cc:3037] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:26.848407 21565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c715dc7b8da49449b37addb9f9edbed" candidate_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1061ad47f8c45739a726adae6735e76"
I20240502 01:52:26.848613 21565 raft_consensus.cc:3037] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:26.851217 21414 raft_consensus.cc:2454] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e49fd5bb4f4590bcec214ff91a47cc in term 2.
I20240502 01:52:26.851428 21565 raft_consensus.cc:2454] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e49fd5bb4f4590bcec214ff91a47cc in term 2.
I20240502 01:52:26.851614 21453 leader_election.cc:302] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e49fd5bb4f4590bcec214ff91a47cc, cee5a47493b545068259cf2cba1ce8b9; no voters:
I20240502 01:52:26.851935 21658 raft_consensus.cc:2781] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:26.852082 21658 raft_consensus.cc:686] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 LEADER]: Becoming Leader. State: Replica: 54e49fd5bb4f4590bcec214ff91a47cc, State: Running, Role: LEADER
I20240502 01:52:26.852348 21658 consensus_queue.cc:237] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } }
I20240502 01:52:26.854002 21312 catalog_manager.cc:5484] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc reported cstate change: term changed from 1 to 2, leader changed from cee5a47493b545068259cf2cba1ce8b9 (127.20.65.65) to 54e49fd5bb4f4590bcec214ff91a47cc (127.20.65.66). New cstate: current_term: 2 leader_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:26.991639 21656 consensus_queue.cc:1035] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:26.995571 21656 consensus_queue.cc:1035] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:27.031944 21656 consensus_queue.cc:1035] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:27.035885 21656 consensus_queue.cc:1035] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:27.158068 21414 raft_consensus.cc:1261] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 2 FOLLOWER]: Refusing update from remote peer 54e49fd5bb4f4590bcec214ff91a47cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:52:27.158730 21658 consensus_queue.cc:1035] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:27.161808 21565 raft_consensus.cc:3037] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:27.164837 21565 raft_consensus.cc:1261] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76 [term 2 FOLLOWER]: Refusing update from remote peer 54e49fd5bb4f4590bcec214ff91a47cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:52:27.165271 21658 consensus_queue.cc:1035] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:27.194784 21565 raft_consensus.cc:1261] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76 [term 2 FOLLOWER]: Refusing update from remote peer 54e49fd5bb4f4590bcec214ff91a47cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:52:27.195266 21658 consensus_queue.cc:1035] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:27.198379 21414 raft_consensus.cc:1261] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 2 FOLLOWER]: Refusing update from remote peer 54e49fd5bb4f4590bcec214ff91a47cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:52:27.198951 21621 consensus_queue.cc:1035] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:27.309563 21565 raft_consensus.cc:1261] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76 [term 2 FOLLOWER]: Refusing update from remote peer 54e49fd5bb4f4590bcec214ff91a47cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:52:27.310091 21621 consensus_queue.cc:1035] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:27.313361 21414 raft_consensus.cc:1261] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 2 FOLLOWER]: Refusing update from remote peer 54e49fd5bb4f4590bcec214ff91a47cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:52:27.313994 21760 consensus_queue.cc:1035] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:27.691167 21736 heartbeater.cc:503] Master 127.20.65.126:34393 was elected leader, sending a full tablet report...
I20240502 01:52:28.755542 21365 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:52:28.757532 21365 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:52:28.757695 21365 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:52:30.758059 21365 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:52:30.759485 21365 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:52:30.759670 21365 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:52:31.750283 21414 consensus_queue.cc:237] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } }
I20240502 01:52:31.752777 21489 raft_consensus.cc:1261] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Refusing update from remote peer cee5a47493b545068259cf2cba1ce8b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:31.753587 21653 consensus_queue.cc:1035] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:31.756245 21565 raft_consensus.cc:1261] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Refusing update from remote peer cee5a47493b545068259cf2cba1ce8b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:31.757448 21762 consensus_queue.cc:1035] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:31.760818 21762 raft_consensus.cc:2932] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58b53c132697421181f99cda363311ef (127.20.65.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } } }
I20240502 01:52:31.761580 21489 raft_consensus.cc:2932] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58b53c132697421181f99cda363311ef (127.20.65.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } } }
I20240502 01:52:31.767556 21312 catalog_manager.cc:5484] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 reported cstate change: config changed from index -1 to 2, NON_VOTER 58b53c132697421181f99cda363311ef (127.20.65.68) added. New cstate: current_term: 1 leader_uuid: "cee5a47493b545068259cf2cba1ce8b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:31.769050 21377 consensus_peers.cc:485] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 -> Peer 58b53c132697421181f99cda363311ef (127.20.65.68:35677): Couldn't send request to peer 58b53c132697421181f99cda363311ef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3eef96a60b4145f4b5b677cfa6c91aeb. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:31.772305 21564 raft_consensus.cc:2932] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58b53c132697421181f99cda363311ef (127.20.65.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } } }
I20240502 01:52:31.776150 21565 consensus_queue.cc:237] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } attrs { replace: true } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } }
I20240502 01:52:31.786926 21414 raft_consensus.cc:1261] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Refusing update from remote peer d1061ad47f8c45739a726adae6735e76: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:31.787689 21489 raft_consensus.cc:1261] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Refusing update from remote peer d1061ad47f8c45739a726adae6735e76: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:31.787555 21777 consensus_queue.cc:1035] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:31.788214 21777 consensus_queue.cc:1035] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:31.790531 21647 raft_consensus.cc:2932] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58b53c132697421181f99cda363311ef (127.20.65.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } attrs { replace: true } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } } }
I20240502 01:52:31.791185 21414 raft_consensus.cc:2932] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58b53c132697421181f99cda363311ef (127.20.65.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } attrs { replace: true } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } } }
I20240502 01:52:31.794950 21489 raft_consensus.cc:2932] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58b53c132697421181f99cda363311ef (127.20.65.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } attrs { replace: true } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } } }
I20240502 01:52:31.795432 21310 catalog_manager.cc:5484] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 reported cstate change: config changed from index -1 to 2, NON_VOTER 58b53c132697421181f99cda363311ef (127.20.65.68) added. New cstate: current_term: 1 leader_uuid: "d1061ad47f8c45739a726adae6735e76" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:31.796447 21413 consensus_queue.cc:237] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } }
W20240502 01:52:31.800587 21527 consensus_peers.cc:485] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 -> Peer 58b53c132697421181f99cda363311ef (127.20.65.68:35677): Couldn't send request to peer 58b53c132697421181f99cda363311ef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2cca93e3d3e64bd4bb2f7dbdc9adbdb8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:31.801093 21565 raft_consensus.cc:1261] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Refusing update from remote peer cee5a47493b545068259cf2cba1ce8b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:52:31.801456 21377 consensus_peers.cc:485] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 -> Peer 58b53c132697421181f99cda363311ef (127.20.65.68:35677): Couldn't send request to peer 58b53c132697421181f99cda363311ef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd97875d904c412cbb071e82158bb046. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:31.801504 21489 raft_consensus.cc:1261] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Refusing update from remote peer cee5a47493b545068259cf2cba1ce8b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:31.801632 21762 consensus_queue.cc:1035] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:31.802139 21653 consensus_queue.cc:1035] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:31.804244 21762 raft_consensus.cc:2932] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58b53c132697421181f99cda363311ef (127.20.65.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } } }
I20240502 01:52:31.804859 21565 raft_consensus.cc:2932] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58b53c132697421181f99cda363311ef (127.20.65.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } } }
I20240502 01:52:31.805536 21489 raft_consensus.cc:2932] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58b53c132697421181f99cda363311ef (127.20.65.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } } }
I20240502 01:52:31.809389 21311 catalog_manager.cc:5484] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 reported cstate change: config changed from index -1 to 2, NON_VOTER 58b53c132697421181f99cda363311ef (127.20.65.68) added. New cstate: current_term: 1 leader_uuid: "cee5a47493b545068259cf2cba1ce8b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } } }
I20240502 01:52:31.810355 21413 consensus_queue.cc:237] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } }
I20240502 01:52:31.813113 21565 raft_consensus.cc:1261] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Refusing update from remote peer cee5a47493b545068259cf2cba1ce8b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:31.813601 21489 raft_consensus.cc:1261] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Refusing update from remote peer cee5a47493b545068259cf2cba1ce8b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:31.813669 21762 consensus_queue.cc:1035] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:52:31.813149 21377 consensus_peers.cc:485] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 -> Peer 58b53c132697421181f99cda363311ef (127.20.65.68:35677): Couldn't send request to peer 58b53c132697421181f99cda363311ef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c9c38df1bb7425d8bb9ef4fe003cd3a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:31.814194 21653 consensus_queue.cc:1035] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:31.816711 21762 raft_consensus.cc:2932] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58b53c132697421181f99cda363311ef (127.20.65.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } } }
I20240502 01:52:31.817337 21565 raft_consensus.cc:2932] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58b53c132697421181f99cda363311ef (127.20.65.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } } }
I20240502 01:52:31.817873 21489 raft_consensus.cc:2932] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58b53c132697421181f99cda363311ef (127.20.65.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } } }
I20240502 01:52:31.821669 21310 catalog_manager.cc:5484] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76 reported cstate change: config changed from index -1 to 2, NON_VOTER 58b53c132697421181f99cda363311ef (127.20.65.68) added. New cstate: current_term: 1 leader_uuid: "cee5a47493b545068259cf2cba1ce8b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cee5a47493b545068259cf2cba1ce8b9" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 35787 } } peers { permanent_uuid: "54e49fd5bb4f4590bcec214ff91a47cc" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d1061ad47f8c45739a726adae6735e76" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 38835 } } peers { permanent_uuid: "58b53c132697421181f99cda363311ef" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 35677 } attrs { promote: true } } }
I20240502 01:52:31.938187 20741 tablet_server.cc:178] TabletServer@127.20.65.65:0 shutting down...
W20240502 01:52:31.940572 21295 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20240502 01:52:31.942735 21360 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 01:52:31.942842 21360 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 01:52:31.945124 21360 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 01:52:31.945233 21360 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 01:52:31.946389 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:52:31.946710 20741 tablet_replica.cc:330] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9: stopping tablet replica
I20240502 01:52:31.946971 20741 raft_consensus.cc:2229] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:52:31.947505 20741 raft_consensus.cc:2258] T 3eef96a60b4145f4b5b677cfa6c91aeb P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:31.948318 20741 tablet_replica.cc:330] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9: stopping tablet replica
W20240502 01:52:31.948433 21360 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 01:52:31.948518 21360 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 01:52:31.948545 20741 raft_consensus.cc:2229] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:52:31.948922 20741 raft_consensus.cc:2258] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:31.949581 20741 tablet_replica.cc:330] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9: stopping tablet replica
I20240502 01:52:31.949788 20741 raft_consensus.cc:2229] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:31.949980 20741 raft_consensus.cc:2258] T 225d0d79a5944db997b85fc3e4578893 P cee5a47493b545068259cf2cba1ce8b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:31.950534 20741 tablet_replica.cc:330] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9: stopping tablet replica
I20240502 01:52:31.950695 20741 raft_consensus.cc:2229] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:31.950875 20741 raft_consensus.cc:2258] T 1db6e77e52ff488b9631d9093ee2f78f P cee5a47493b545068259cf2cba1ce8b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:31.951457 20741 tablet_replica.cc:330] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9: stopping tablet replica
I20240502 01:52:31.951613 20741 raft_consensus.cc:2229] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:52:31.951926 20741 raft_consensus.cc:2258] T bd97875d904c412cbb071e82158bb046 P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:31.952575 20741 tablet_replica.cc:330] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9: stopping tablet replica
I20240502 01:52:31.952736 20741 raft_consensus.cc:2229] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:31.952903 20741 raft_consensus.cc:2258] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P cee5a47493b545068259cf2cba1ce8b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:31.953601 20741 tablet_replica.cc:330] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9: stopping tablet replica
I20240502 01:52:31.953771 20741 raft_consensus.cc:2229] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:31.953963 20741 raft_consensus.cc:2258] T 280305ca0fa14695b45a76e78cfff168 P cee5a47493b545068259cf2cba1ce8b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:31.954504 20741 tablet_replica.cc:330] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9: stopping tablet replica
I20240502 01:52:31.954675 20741 raft_consensus.cc:2229] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:31.954842 20741 raft_consensus.cc:2258] T 6c715dc7b8da49449b37addb9f9edbed P cee5a47493b545068259cf2cba1ce8b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:31.972244 20741 tablet_server.cc:194] TabletServer@127.20.65.65:0 shutdown complete.
I20240502 01:52:31.977270 20741 tablet_server.cc:178] TabletServer@127.20.65.66:0 shutting down...
I20240502 01:52:31.986969 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:52:31.987223 20741 tablet_replica.cc:330] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc: stopping tablet replica
I20240502 01:52:31.987573 20741 raft_consensus.cc:2229] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:31.987812 20741 raft_consensus.cc:2258] T bd97875d904c412cbb071e82158bb046 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:31.988655 20741 tablet_replica.cc:330] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc: stopping tablet replica
I20240502 01:52:31.988831 20741 raft_consensus.cc:2229] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:31.989048 20741 raft_consensus.cc:2258] T 3eef96a60b4145f4b5b677cfa6c91aeb P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:31.989712 20741 tablet_replica.cc:330] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc: stopping tablet replica
I20240502 01:52:31.989904 20741 raft_consensus.cc:2229] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 LEADER]: Raft consensus shutting down.
I20240502 01:52:31.990229 20741 raft_consensus.cc:2258] T 1db6e77e52ff488b9631d9093ee2f78f P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:31.990760 20741 tablet_replica.cc:330] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc: stopping tablet replica
I20240502 01:52:31.990896 20741 raft_consensus.cc:2229] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:31.991104 20741 raft_consensus.cc:2258] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:31.991775 20741 tablet_replica.cc:330] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc: stopping tablet replica
I20240502 01:52:31.991927 20741 raft_consensus.cc:2229] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 LEADER]: Raft consensus shutting down.
I20240502 01:52:31.992246 20741 raft_consensus.cc:2258] T 225d0d79a5944db997b85fc3e4578893 P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:31.992772 20741 tablet_replica.cc:330] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc: stopping tablet replica
I20240502 01:52:31.992921 20741 raft_consensus.cc:2229] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:31.993083 20741 raft_consensus.cc:2258] T 280305ca0fa14695b45a76e78cfff168 P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:31.993597 20741 tablet_replica.cc:330] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc: stopping tablet replica
I20240502 01:52:31.993782 20741 raft_consensus.cc:2229] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:31.993974 20741 raft_consensus.cc:2258] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P 54e49fd5bb4f4590bcec214ff91a47cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:31.994742 20741 tablet_replica.cc:330] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc: stopping tablet replica
I20240502 01:52:31.994900 20741 raft_consensus.cc:2229] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 LEADER]: Raft consensus shutting down.
I20240502 01:52:31.995188 20741 raft_consensus.cc:2258] T 6c715dc7b8da49449b37addb9f9edbed P 54e49fd5bb4f4590bcec214ff91a47cc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:32.011121 20741 tablet_server.cc:194] TabletServer@127.20.65.66:0 shutdown complete.
I20240502 01:52:32.016019 20741 tablet_server.cc:178] TabletServer@127.20.65.67:0 shutting down...
W20240502 01:52:32.019423 21360 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 01:52:32.019543 21360 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 01:52:32.025394 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:52:32.025629 20741 tablet_replica.cc:330] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76: stopping tablet replica
I20240502 01:52:32.025810 20741 raft_consensus.cc:2229] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:32.026012 20741 raft_consensus.cc:2258] T 3eef96a60b4145f4b5b677cfa6c91aeb P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:32.026684 20741 tablet_replica.cc:330] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76: stopping tablet replica
I20240502 01:52:32.026859 20741 raft_consensus.cc:2229] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:32.027029 20741 raft_consensus.cc:2258] T 1c9c38df1bb7425d8bb9ef4fe003cd3a P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:32.027612 20741 tablet_replica.cc:330] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76: stopping tablet replica
I20240502 01:52:32.027751 20741 raft_consensus.cc:2229] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 2 LEADER]: Raft consensus shutting down.
I20240502 01:52:32.028012 20741 raft_consensus.cc:2258] T 280305ca0fa14695b45a76e78cfff168 P d1061ad47f8c45739a726adae6735e76 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:32.028417 20741 tablet_replica.cc:330] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76: stopping tablet replica
I20240502 01:52:32.028566 20741 raft_consensus.cc:2229] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:32.028707 20741 raft_consensus.cc:2258] T 1db6e77e52ff488b9631d9093ee2f78f P d1061ad47f8c45739a726adae6735e76 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:32.029135 20741 tablet_replica.cc:330] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76: stopping tablet replica
I20240502 01:52:32.029274 20741 raft_consensus.cc:2229] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:32.029465 20741 raft_consensus.cc:2258] T bd97875d904c412cbb071e82158bb046 P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:32.030010 20741 tablet_replica.cc:330] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76: stopping tablet replica
I20240502 01:52:32.030158 20741 raft_consensus.cc:2229] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:52:32.030499 20741 raft_consensus.cc:2258] T 2cca93e3d3e64bd4bb2f7dbdc9adbdb8 P d1061ad47f8c45739a726adae6735e76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:32.031049 20741 tablet_replica.cc:330] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76: stopping tablet replica
I20240502 01:52:32.031177 20741 raft_consensus.cc:2229] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:32.031332 20741 raft_consensus.cc:2258] T 6c715dc7b8da49449b37addb9f9edbed P d1061ad47f8c45739a726adae6735e76 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:32.031729 20741 tablet_replica.cc:330] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76: stopping tablet replica
I20240502 01:52:32.031873 20741 raft_consensus.cc:2229] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:32.032013 20741 raft_consensus.cc:2258] T 225d0d79a5944db997b85fc3e4578893 P d1061ad47f8c45739a726adae6735e76 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:32.047084 20741 tablet_server.cc:194] TabletServer@127.20.65.67:0 shutdown complete.
I20240502 01:52:32.050952 20741 tablet_server.cc:178] TabletServer@127.20.65.68:0 shutting down...
I20240502 01:52:32.057859 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:52:32.070670 20741 tablet_server.cc:194] TabletServer@127.20.65.68:0 shutdown complete.
I20240502 01:52:32.073202 20741 master.cc:537] Master@127.20.65.126:34393 shutting down...
I20240502 01:52:32.079649 20741 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:52:32.079854 20741 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:32.079958 20741 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7652667f8dda40d8b6bdba0930be5092: stopping tablet replica
I20240502 01:52:32.093242 20741 master.cc:552] Master@127.20.65.126:34393 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7427 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240502 01:52:32.110270 20741 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.65.126:36605,127.20.65.125:39035,127.20.65.124:38337
I20240502 01:52:32.110875 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:32.113657 21788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.114575 21789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.115190 21791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.116153 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:32.116423 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:32.116492 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:32.116570 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614752116533 us; error 0 us; skew 500 ppm
I20240502 01:52:32.116729 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:32.118019 20741 webserver.cc:460] Webserver started at http://127.20.65.126:45151/ using document root <none> and password file <none>
I20240502 01:52:32.118244 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:32.118338 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:32.118497 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:32.119148 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-0-root/instance:
uuid: "7b5a257cabd146be941293466a8fbed1"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-7cj2"
I20240502 01:52:32.121613 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:52:32.123250 21796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.123648 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:52:32.123746 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-0-root
uuid: "7b5a257cabd146be941293466a8fbed1"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-7cj2"
I20240502 01:52:32.123865 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:32.145259 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:32.145790 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:32.158107 21849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.126:36605 every 8 connection(s)
I20240502 01:52:32.158087 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.126:36605
I20240502 01:52:32.159480 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:52:32.159579 21850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:52:32.162518 21852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.162978 21853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.163393 21850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:52:32.163858 20741 server_base.cc:1034] running on GCE node
W20240502 01:52:32.164779 21855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.165237 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:32.165354 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:32.165467 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614752165406 us; error 0 us; skew 500 ppm
I20240502 01:52:32.165688 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:32.167162 20741 webserver.cc:460] Webserver started at http://127.20.65.125:38651/ using document root <none> and password file <none>
I20240502 01:52:32.167398 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:32.167497 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:32.167618 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:32.168295 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-1-root/instance:
uuid: "7f6298ac171e46209830ff574513c63a"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-7cj2"
I20240502 01:52:32.170882 21850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:52:32.171247 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
W20240502 01:52:32.172796 21850 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.65.125:39035: Network error: Client connection negotiation failed: client connection to 127.20.65.125:39035: connect: Connection refused (error 111)
I20240502 01:52:32.172940 21863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.173256 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 01:52:32.173357 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-1-root
uuid: "7f6298ac171e46209830ff574513c63a"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-7cj2"
I20240502 01:52:32.173488 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:32.186888 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:32.187419 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:32.199688 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.125:39035
I20240502 01:52:32.199725 21915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.125:39035 every 8 connection(s)
I20240502 01:52:32.200959 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:52:32.201084 21916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:52:32.204162 21918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.204670 21919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.205447 21916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } has no permanent_uuid. Determining permanent_uuid...
W20240502 01:52:32.205632 21921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.206288 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:32.206682 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:32.206768 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:32.206878 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614752206838 us; error 0 us; skew 500 ppm
I20240502 01:52:32.207090 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:32.208595 20741 webserver.cc:460] Webserver started at http://127.20.65.124:43547/ using document root <none> and password file <none>
I20240502 01:52:32.208844 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:32.208950 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:32.209077 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:32.209779 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-2-root/instance:
uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-7cj2"
I20240502 01:52:32.212144 21916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:52:32.212791 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 01:52:32.214758 21928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.215107 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240502 01:52:32.215222 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-2-root
uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-7cj2"
I20240502 01:52:32.215370 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:32.219072 21916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } has no permanent_uuid. Determining permanent_uuid...
W20240502 01:52:32.220957 21916 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.65.124:38337: Network error: Client connection negotiation failed: client connection to 127.20.65.124:38337: connect: Connection refused (error 111)
I20240502 01:52:32.227416 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:32.227910 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:32.228003 21850 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } attempt: 1
I20240502 01:52:32.234594 21850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } has no permanent_uuid. Determining permanent_uuid...
W20240502 01:52:32.236384 21850 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.65.124:38337: Network error: Client connection negotiation failed: client connection to 127.20.65.124:38337: connect: Connection refused (error 111)
I20240502 01:52:32.238174 21916 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } attempt: 1
W20240502 01:52:32.239861 21916 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.65.124:38337: Network error: Client connection negotiation failed: client connection to 127.20.65.124:38337: connect: Connection refused (error 111)
I20240502 01:52:32.243294 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.124:38337
I20240502 01:52:32.243332 21981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.124:38337 every 8 connection(s)
I20240502 01:52:32.244185 20741 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:52:32.244473 21982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:32.246709 21982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:52:32.251859 21982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:52:32.256505 21982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:52:32.263788 21982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f: Bootstrap starting.
I20240502 01:52:32.264587 21850 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } attempt: 1
I20240502 01:52:32.265187 21982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:32.266958 21982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f: No bootstrap required, opened a new log
I20240502 01:52:32.267597 21982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } }
I20240502 01:52:32.267789 21982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:32.267899 21982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fdf79bd9d594be1942fbeef1d6d2c8f, State: Initialized, Role: FOLLOWER
I20240502 01:52:32.268170 21982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } }
I20240502 01:52:32.268857 21987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } } }
I20240502 01:52:32.269198 21982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:32.269263 21987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:52:32.272328 21850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1: Bootstrap starting.
I20240502 01:52:32.273926 21850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1: Neither blocks nor log segments found. Creating new log.
W20240502 01:52:32.274765 21999 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:52:32.274866 21999 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 01:52:32.275663 21850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1: No bootstrap required, opened a new log
I20240502 01:52:32.276204 21850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } }
I20240502 01:52:32.276373 21850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:32.276454 21850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5a257cabd146be941293466a8fbed1, State: Initialized, Role: FOLLOWER
I20240502 01:52:32.276683 21850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } }
I20240502 01:52:32.277247 22001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } } }
I20240502 01:52:32.277428 21850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:32.277459 22001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:52:32.280814 20741 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240502 01:52:32.281596 22012 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:52:32.281682 22012 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 01:52:32.304062 21916 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } attempt: 2
I20240502 01:52:32.311859 21916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a: Bootstrap starting.
I20240502 01:52:32.313055 21916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:32.314679 21916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a: No bootstrap required, opened a new log
I20240502 01:52:32.315259 21916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } }
I20240502 01:52:32.315424 21916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:32.315479 21916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6298ac171e46209830ff574513c63a, State: Initialized, Role: FOLLOWER
I20240502 01:52:32.315719 21916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } }
I20240502 01:52:32.316304 22014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } } }
I20240502 01:52:32.316439 21916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:32.316529 22014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:52:32.320241 20741 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240502 01:52:32.321072 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:32.321460 22025 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:52:32.321578 22025 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 01:52:32.324693 22026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.325420 22027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.326216 22029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.326957 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:32.327173 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:32.327239 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:32.327279 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614752327266 us; error 0 us; skew 500 ppm
I20240502 01:52:32.327428 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:32.328636 20741 webserver.cc:460] Webserver started at http://127.20.65.65:32809/ using document root <none> and password file <none>
I20240502 01:52:32.328856 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:32.328922 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:32.329061 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:32.329639 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-0-root/instance:
uuid: "8342461f3c1d4930a93e5eac1971b6f1"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-7cj2"
I20240502 01:52:32.332121 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:52:32.333660 22034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.333997 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 01:52:32.334106 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-0-root
uuid: "8342461f3c1d4930a93e5eac1971b6f1"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-7cj2"
I20240502 01:52:32.334220 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:32.344563 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:32.345232 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:32.345844 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:32.346812 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:32.346916 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.347023 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:32.347095 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.362437 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.65:42293
I20240502 01:52:32.362478 22096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.65:42293 every 8 connection(s)
I20240502 01:52:32.365548 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:52:32.376123 22098 heartbeater.cc:347] Connected to a master server at 127.20.65.126:36605
I20240502 01:52:32.376344 22098 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:32.376523 22097 heartbeater.cc:347] Connected to a master server at 127.20.65.124:38337
I20240502 01:52:32.376721 22097 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:32.377084 22098 heartbeater.cc:511] Master 127.20.65.126:36605 requested a full tablet report, sending...
I20240502 01:52:32.377406 22097 heartbeater.cc:511] Master 127.20.65.124:38337 requested a full tablet report, sending...
W20240502 01:52:32.377910 22104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.378952 21814 ts_manager.cc:194] Registered new tserver with Master: 8342461f3c1d4930a93e5eac1971b6f1 (127.20.65.65:42293)
W20240502 01:52:32.379969 22105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.380789 21947 ts_manager.cc:194] Registered new tserver with Master: 8342461f3c1d4930a93e5eac1971b6f1 (127.20.65.65:42293)
W20240502 01:52:32.383133 22107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.383153 22099 heartbeater.cc:347] Connected to a master server at 127.20.65.125:39035
I20240502 01:52:32.383368 22099 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:32.383699 22099 heartbeater.cc:511] Master 127.20.65.125:39035 requested a full tablet report, sending...
I20240502 01:52:32.383903 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:32.384196 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:32.384276 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:32.384351 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614752384314 us; error 0 us; skew 500 ppm
I20240502 01:52:32.384536 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:32.384694 21880 ts_manager.cc:194] Registered new tserver with Master: 8342461f3c1d4930a93e5eac1971b6f1 (127.20.65.65:42293)
I20240502 01:52:32.385943 20741 webserver.cc:460] Webserver started at http://127.20.65.66:32867/ using document root <none> and password file <none>
I20240502 01:52:32.386154 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:32.386250 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:32.386385 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:32.387050 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-1-root/instance:
uuid: "80ad8708a5cd4a80bd711523383e7ea1"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-7cj2"
I20240502 01:52:32.389422 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:52:32.391057 22112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.391420 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240502 01:52:32.391531 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-1-root
uuid: "80ad8708a5cd4a80bd711523383e7ea1"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-7cj2"
I20240502 01:52:32.391654 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:32.405087 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:32.405613 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:32.406136 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:32.407016 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:32.407089 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.407202 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:32.407258 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.424273 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.66:39321
I20240502 01:52:32.424321 22176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.66:39321 every 8 connection(s)
I20240502 01:52:32.438031 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:52:32.441368 22178 heartbeater.cc:347] Connected to a master server at 127.20.65.126:36605
I20240502 01:52:32.441502 22177 heartbeater.cc:347] Connected to a master server at 127.20.65.124:38337
I20240502 01:52:32.441586 22178 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:32.441648 22177 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:32.442183 22177 heartbeater.cc:511] Master 127.20.65.124:38337 requested a full tablet report, sending...
I20240502 01:52:32.442308 22178 heartbeater.cc:511] Master 127.20.65.126:36605 requested a full tablet report, sending...
I20240502 01:52:32.443323 21947 ts_manager.cc:194] Registered new tserver with Master: 80ad8708a5cd4a80bd711523383e7ea1 (127.20.65.66:39321)
I20240502 01:52:32.443727 21814 ts_manager.cc:194] Registered new tserver with Master: 80ad8708a5cd4a80bd711523383e7ea1 (127.20.65.66:39321)
I20240502 01:52:32.444989 22179 heartbeater.cc:347] Connected to a master server at 127.20.65.125:39035
I20240502 01:52:32.445183 22179 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:32.445734 22179 heartbeater.cc:511] Master 127.20.65.125:39035 requested a full tablet report, sending...
I20240502 01:52:32.446858 21880 ts_manager.cc:194] Registered new tserver with Master: 80ad8708a5cd4a80bd711523383e7ea1 (127.20.65.66:39321)
W20240502 01:52:32.448087 22184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.448628 22185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.449445 22187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.452050 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:32.452291 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:32.452375 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:32.452441 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614752452421 us; error 0 us; skew 500 ppm
I20240502 01:52:32.452617 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:32.453837 20741 webserver.cc:460] Webserver started at http://127.20.65.67:34587/ using document root <none> and password file <none>
I20240502 01:52:32.454080 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:32.454169 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:32.454316 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:32.454946 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-2-root/instance:
uuid: "2ca785ff912d4397a4dc6eec2c9ce7aa"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-7cj2"
I20240502 01:52:32.457258 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:32.458847 22192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.459231 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240502 01:52:32.459324 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-2-root
uuid: "2ca785ff912d4397a4dc6eec2c9ce7aa"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-7cj2"
I20240502 01:52:32.459414 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614741486931-20741-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:32.467566 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:32.468055 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:32.468583 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:32.469434 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:32.469514 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.469599 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:32.469687 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:32.486568 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.67:44307
I20240502 01:52:32.486608 22255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.67:44307 every 8 connection(s)
I20240502 01:52:32.497185 22257 heartbeater.cc:347] Connected to a master server at 127.20.65.126:36605
I20240502 01:52:32.497210 22256 heartbeater.cc:347] Connected to a master server at 127.20.65.124:38337
I20240502 01:52:32.497386 22257 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:32.497440 22256 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:32.498023 22256 heartbeater.cc:511] Master 127.20.65.124:38337 requested a full tablet report, sending...
I20240502 01:52:32.498051 22257 heartbeater.cc:511] Master 127.20.65.126:36605 requested a full tablet report, sending...
I20240502 01:52:32.498656 22258 heartbeater.cc:347] Connected to a master server at 127.20.65.125:39035
I20240502 01:52:32.498809 22258 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:32.499089 21947 ts_manager.cc:194] Registered new tserver with Master: 2ca785ff912d4397a4dc6eec2c9ce7aa (127.20.65.67:44307)
I20240502 01:52:32.499279 22258 heartbeater.cc:511] Master 127.20.65.125:39035 requested a full tablet report, sending...
I20240502 01:52:32.499715 21814 ts_manager.cc:194] Registered new tserver with Master: 2ca785ff912d4397a4dc6eec2c9ce7aa (127.20.65.67:44307)
I20240502 01:52:32.500276 21880 ts_manager.cc:194] Registered new tserver with Master: 2ca785ff912d4397a4dc6eec2c9ce7aa (127.20.65.67:44307)
I20240502 01:52:32.500326 20741 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.009241805s
I20240502 01:52:32.677289 21987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:32.677527 21987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } }
I20240502 01:52:32.678213 21987 leader_election.cc:288] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b5a257cabd146be941293466a8fbed1 (127.20.65.126:36605), 7f6298ac171e46209830ff574513c63a (127.20.65.125:39035)
I20240502 01:52:32.678820 21824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5a257cabd146be941293466a8fbed1" is_pre_election: true
I20240502 01:52:32.678893 21890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6298ac171e46209830ff574513c63a" is_pre_election: true
I20240502 01:52:32.679054 21824 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fdf79bd9d594be1942fbeef1d6d2c8f in term 0.
I20240502 01:52:32.679122 21890 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fdf79bd9d594be1942fbeef1d6d2c8f in term 0.
I20240502 01:52:32.679502 21929 leader_election.cc:302] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b5a257cabd146be941293466a8fbed1, 7fdf79bd9d594be1942fbeef1d6d2c8f; no voters:
I20240502 01:52:32.679783 21987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:32.679893 21987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:32.680001 21987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:32.681980 21987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } }
I20240502 01:52:32.682566 21987 leader_election.cc:288] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [CANDIDATE]: Term 1 election: Requested vote from peers 7b5a257cabd146be941293466a8fbed1 (127.20.65.126:36605), 7f6298ac171e46209830ff574513c63a (127.20.65.125:39035)
I20240502 01:52:32.683045 21824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5a257cabd146be941293466a8fbed1"
I20240502 01:52:32.683125 21890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6298ac171e46209830ff574513c63a"
I20240502 01:52:32.683239 21824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:32.683300 21890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:32.685310 21824 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fdf79bd9d594be1942fbeef1d6d2c8f in term 1.
I20240502 01:52:32.685333 21890 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fdf79bd9d594be1942fbeef1d6d2c8f in term 1.
I20240502 01:52:32.685729 21929 leader_election.cc:302] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f6298ac171e46209830ff574513c63a, 7fdf79bd9d594be1942fbeef1d6d2c8f; no voters:
I20240502 01:52:32.686008 21987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:32.686374 21987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [term 1 LEADER]: Becoming Leader. State: Replica: 7fdf79bd9d594be1942fbeef1d6d2c8f, State: Running, Role: LEADER
I20240502 01:52:32.686674 21987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } }
I20240502 01:52:32.688019 22264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fdf79bd9d594be1942fbeef1d6d2c8f. Latest consensus state: current_term: 1 leader_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } } }
I20240502 01:52:32.688259 22264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:32.688928 22267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:32.690248 22267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:32.693087 21824 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 1 FOLLOWER]: Refusing update from remote peer 7fdf79bd9d594be1942fbeef1d6d2c8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:32.693105 21890 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 1 FOLLOWER]: Refusing update from remote peer 7fdf79bd9d594be1942fbeef1d6d2c8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:32.693650 22264 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:32.693864 21987 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:32.700428 22014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fdf79bd9d594be1942fbeef1d6d2c8f. Latest consensus state: current_term: 1 leader_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } } }
I20240502 01:52:32.700767 22014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:52:32.701303 22001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fdf79bd9d594be1942fbeef1d6d2c8f. Latest consensus state: current_term: 1 leader_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } } }
I20240502 01:52:32.701529 22001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:52:32.703557 22264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } } }
I20240502 01:52:32.703617 21987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } } }
I20240502 01:52:32.703879 21987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:32.703912 22001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } } }
I20240502 01:52:32.703878 22264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:32.704373 22001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:52:32.705001 22267 catalog_manager.cc:1347] Generated new cluster ID: ba1241031aeb4ec590fda072ecb5c87e
I20240502 01:52:32.705073 22267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:32.707365 22014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } } }
I20240502 01:52:32.707576 22014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:52:32.730448 22267 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:32.731395 22267 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:32.739111 22267 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f: Generated new TSK 0
I20240502 01:52:32.739454 22267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:33.282941 22012 catalog_manager.cc:1259] Loaded cluster ID: ba1241031aeb4ec590fda072ecb5c87e
I20240502 01:52:33.283046 22012 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1: loading cluster ID for follower catalog manager: success
I20240502 01:52:33.284973 22012 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1: acquiring CA information for follower catalog manager: success
I20240502 01:52:33.285951 22012 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 01:52:33.322865 22025 catalog_manager.cc:1259] Loaded cluster ID: ba1241031aeb4ec590fda072ecb5c87e
I20240502 01:52:33.322960 22025 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a: loading cluster ID for follower catalog manager: success
I20240502 01:52:33.325125 22025 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a: acquiring CA information for follower catalog manager: success
I20240502 01:52:33.326167 22025 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 01:52:33.382983 21947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33728
I20240502 01:52:33.446074 21947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33732
I20240502 01:52:33.502076 21947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33748
I20240502 01:52:34.273766 21997 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:52:34.354789 20741 test_util.cc:268] Using random seed: -614717871
I20240502 01:52:34.384179 22097 heartbeater.cc:503] Master 127.20.65.124:38337 was elected leader, sending a full tablet report...
I20240502 01:52:34.386855 21947 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33752:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 01:52:34.410763 22062 tablet_service.cc:1467] Processing CreateTablet for tablet ccbde4d521a5459f8d1433cc421324c5 (DEFAULT_TABLE table=test-workload [id=9cf115c385ee431484aaa097921b39ba]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 01:52:34.411365 22062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccbde4d521a5459f8d1433cc421324c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:34.420269 22220 tablet_service.cc:1467] Processing CreateTablet for tablet a11207a1191b4774a90ad5291de4368c (DEFAULT_TABLE table=test-workload [id=9cf115c385ee431484aaa097921b39ba]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 01:52:34.420480 22142 tablet_service.cc:1467] Processing CreateTablet for tablet 2b79da6d94c54f99ae9b3f0b066e4825 (DEFAULT_TABLE table=test-workload [id=9cf115c385ee431484aaa097921b39ba]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 01:52:34.420717 22220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a11207a1191b4774a90ad5291de4368c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:34.421070 22142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b79da6d94c54f99ae9b3f0b066e4825. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:34.425773 22306 tablet_bootstrap.cc:492] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1: Bootstrap starting.
I20240502 01:52:34.426086 22307 tablet_bootstrap.cc:492] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa: Bootstrap starting.
I20240502 01:52:34.427930 22307 tablet_bootstrap.cc:654] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:34.428295 22306 tablet_bootstrap.cc:654] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:34.429944 22308 tablet_bootstrap.cc:492] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1: Bootstrap starting.
I20240502 01:52:34.431355 22308 tablet_bootstrap.cc:654] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:34.433292 22308 tablet_bootstrap.cc:492] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1: No bootstrap required, opened a new log
I20240502 01:52:34.433452 22306 tablet_bootstrap.cc:492] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1: No bootstrap required, opened a new log
I20240502 01:52:34.433477 22308 ts_tablet_manager.cc:1367] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:34.433599 22306 ts_tablet_manager.cc:1367] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 01:52:34.433696 22307 tablet_bootstrap.cc:492] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa: No bootstrap required, opened a new log
I20240502 01:52:34.433843 22307 ts_tablet_manager.cc:1367] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:52:34.434294 22306 raft_consensus.cc:348] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8342461f3c1d4930a93e5eac1971b6f1" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 42293 } }
I20240502 01:52:34.434513 22306 raft_consensus.cc:374] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:34.434569 22307 raft_consensus.cc:348] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca785ff912d4397a4dc6eec2c9ce7aa" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 44307 } }
I20240502 01:52:34.434451 22308 raft_consensus.cc:348] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ad8708a5cd4a80bd711523383e7ea1" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 39321 } }
I20240502 01:52:34.434723 22306 raft_consensus.cc:729] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8342461f3c1d4930a93e5eac1971b6f1, State: Initialized, Role: FOLLOWER
I20240502 01:52:34.434734 22307 raft_consensus.cc:374] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:34.434742 22308 raft_consensus.cc:374] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:34.434923 22307 raft_consensus.cc:729] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca785ff912d4397a4dc6eec2c9ce7aa, State: Initialized, Role: FOLLOWER
I20240502 01:52:34.435014 22308 raft_consensus.cc:729] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ad8708a5cd4a80bd711523383e7ea1, State: Initialized, Role: FOLLOWER
I20240502 01:52:34.434968 22306 consensus_queue.cc:260] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8342461f3c1d4930a93e5eac1971b6f1" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 42293 } }
I20240502 01:52:34.435156 22307 consensus_queue.cc:260] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca785ff912d4397a4dc6eec2c9ce7aa" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 44307 } }
I20240502 01:52:34.435247 22306 raft_consensus.cc:388] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:34.435345 22306 raft_consensus.cc:482] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:34.435350 22307 raft_consensus.cc:388] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:34.435371 22308 consensus_queue.cc:260] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ad8708a5cd4a80bd711523383e7ea1" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 39321 } }
I20240502 01:52:34.435506 22307 raft_consensus.cc:482] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:34.435585 22308 raft_consensus.cc:388] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:34.435631 22307 raft_consensus.cc:3037] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:34.435727 22308 raft_consensus.cc:482] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:34.435896 22308 raft_consensus.cc:3037] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:34.435983 22306 raft_consensus.cc:3037] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:34.438196 22306 raft_consensus.cc:504] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8342461f3c1d4930a93e5eac1971b6f1" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 42293 } }
I20240502 01:52:34.438486 22306 leader_election.cc:302] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8342461f3c1d4930a93e5eac1971b6f1; no voters:
I20240502 01:52:34.438879 22307 raft_consensus.cc:504] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca785ff912d4397a4dc6eec2c9ce7aa" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 44307 } }
I20240502 01:52:34.439183 22312 raft_consensus.cc:2781] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:34.439208 22307 leader_election.cc:302] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ca785ff912d4397a4dc6eec2c9ce7aa; no voters:
I20240502 01:52:34.438879 22308 raft_consensus.cc:504] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ad8708a5cd4a80bd711523383e7ea1" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 39321 } }
I20240502 01:52:34.439021 22306 leader_election.cc:288] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:34.439654 22312 raft_consensus.cc:686] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1 [term 1 LEADER]: Becoming Leader. State: Replica: 8342461f3c1d4930a93e5eac1971b6f1, State: Running, Role: LEADER
I20240502 01:52:34.439718 22308 leader_election.cc:302] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80ad8708a5cd4a80bd711523383e7ea1; no voters:
I20240502 01:52:34.440064 22312 consensus_queue.cc:237] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8342461f3c1d4930a93e5eac1971b6f1" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 42293 } }
I20240502 01:52:34.440986 22308 leader_election.cc:288] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:34.441402 22316 raft_consensus.cc:2781] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:34.443353 22308 ts_tablet_manager.cc:1398] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1: Time spent starting tablet: real 0.010s user 0.003s sys 0.004s
I20240502 01:52:34.443440 22314 raft_consensus.cc:2781] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:34.443631 22177 heartbeater.cc:503] Master 127.20.65.124:38337 was elected leader, sending a full tablet report...
I20240502 01:52:34.443658 22307 leader_election.cc:288] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:34.444381 22316 raft_consensus.cc:686] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1 [term 1 LEADER]: Becoming Leader. State: Replica: 80ad8708a5cd4a80bd711523383e7ea1, State: Running, Role: LEADER
I20240502 01:52:34.445283 22316 consensus_queue.cc:237] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ad8708a5cd4a80bd711523383e7ea1" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 39321 } }
I20240502 01:52:34.446250 22307 ts_tablet_manager.cc:1398] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20240502 01:52:34.446169 21947 catalog_manager.cc:5484] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8342461f3c1d4930a93e5eac1971b6f1 (127.20.65.65). New cstate: current_term: 1 leader_uuid: "8342461f3c1d4930a93e5eac1971b6f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8342461f3c1d4930a93e5eac1971b6f1" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 42293 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:34.446522 22256 heartbeater.cc:503] Master 127.20.65.124:38337 was elected leader, sending a full tablet report...
I20240502 01:52:34.444518 22306 ts_tablet_manager.cc:1398] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20240502 01:52:34.448469 22314 raft_consensus.cc:686] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa [term 1 LEADER]: Becoming Leader. State: Replica: 2ca785ff912d4397a4dc6eec2c9ce7aa, State: Running, Role: LEADER
I20240502 01:52:34.449051 22314 consensus_queue.cc:237] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca785ff912d4397a4dc6eec2c9ce7aa" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 44307 } }
I20240502 01:52:34.463325 21947 catalog_manager.cc:5484] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ca785ff912d4397a4dc6eec2c9ce7aa (127.20.65.67). New cstate: current_term: 1 leader_uuid: "2ca785ff912d4397a4dc6eec2c9ce7aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca785ff912d4397a4dc6eec2c9ce7aa" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 44307 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:34.466825 21946 catalog_manager.cc:5484] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80ad8708a5cd4a80bd711523383e7ea1 (127.20.65.66). New cstate: current_term: 1 leader_uuid: "80ad8708a5cd4a80bd711523383e7ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ad8708a5cd4a80bd711523383e7ea1" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 39321 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:34.486261 20741 master.cc:537] Master@127.20.65.124:38337 shutting down...
I20240502 01:52:34.492818 20741 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:52:34.493172 20741 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:34.493299 20741 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7fdf79bd9d594be1942fbeef1d6d2c8f: stopping tablet replica
I20240502 01:52:34.508117 20741 master.cc:552] Master@127.20.65.124:38337 shutdown complete.
W20240502 01:52:35.465932 22097 heartbeater.cc:644] Failed to heartbeat to 127.20.65.124:38337 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.65.124:38337: connect: Connection refused (error 111)
I20240502 01:52:35.974980 22329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7fdf79bd9d594be1942fbeef1d6d2c8f)
I20240502 01:52:35.974936 22330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7fdf79bd9d594be1942fbeef1d6d2c8f)
I20240502 01:52:35.975252 22330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } }
I20240502 01:52:35.975231 22329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } }
I20240502 01:52:35.976017 22329 leader_election.cc:288] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f6298ac171e46209830ff574513c63a (127.20.65.125:39035), 7fdf79bd9d594be1942fbeef1d6d2c8f (127.20.65.124:38337)
I20240502 01:52:35.975988 22330 leader_election.cc:288] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b5a257cabd146be941293466a8fbed1 (127.20.65.126:36605), 7fdf79bd9d594be1942fbeef1d6d2c8f (127.20.65.124:38337)
I20240502 01:52:35.977155 21890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7b5a257cabd146be941293466a8fbed1" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "7f6298ac171e46209830ff574513c63a" is_pre_election: true
I20240502 01:52:35.977435 21890 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b5a257cabd146be941293466a8fbed1 in term 1.
I20240502 01:52:35.978260 21798 leader_election.cc:302] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b5a257cabd146be941293466a8fbed1, 7f6298ac171e46209830ff574513c63a; no voters:
I20240502 01:52:35.978297 21824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f6298ac171e46209830ff574513c63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "7b5a257cabd146be941293466a8fbed1" is_pre_election: true
I20240502 01:52:35.978534 21824 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f6298ac171e46209830ff574513c63a in term 1.
W20240502 01:52:35.978641 21866 leader_election.cc:334] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7fdf79bd9d594be1942fbeef1d6d2c8f (127.20.65.124:38337): Network error: Client connection negotiation failed: client connection to 127.20.65.124:38337: connect: Connection refused (error 111)
I20240502 01:52:35.978861 22329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 01:52:35.979007 21800 leader_election.cc:334] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7fdf79bd9d594be1942fbeef1d6d2c8f (127.20.65.124:38337): Network error: Client connection negotiation failed: client connection to 127.20.65.124:38337: connect: Connection refused (error 111)
I20240502 01:52:35.979030 21864 leader_election.cc:302] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b5a257cabd146be941293466a8fbed1, 7f6298ac171e46209830ff574513c63a; no voters: 7fdf79bd9d594be1942fbeef1d6d2c8f
I20240502 01:52:35.979061 22329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7fdf79bd9d594be1942fbeef1d6d2c8f)
I20240502 01:52:35.979203 22329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:35.979321 22330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:52:35.979458 22330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7fdf79bd9d594be1942fbeef1d6d2c8f)
I20240502 01:52:35.979565 22330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:35.981498 22329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } }
I20240502 01:52:35.981520 22330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } }
I20240502 01:52:35.982050 22329 leader_election.cc:288] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [CANDIDATE]: Term 2 election: Requested vote from peers 7f6298ac171e46209830ff574513c63a (127.20.65.125:39035), 7fdf79bd9d594be1942fbeef1d6d2c8f (127.20.65.124:38337)
I20240502 01:52:35.982722 21824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f6298ac171e46209830ff574513c63a" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "7b5a257cabd146be941293466a8fbed1"
I20240502 01:52:35.982932 21824 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f6298ac171e46209830ff574513c63a in current term 2: Already voted for candidate 7b5a257cabd146be941293466a8fbed1 in this term.
I20240502 01:52:35.982879 21890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7b5a257cabd146be941293466a8fbed1" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "7f6298ac171e46209830ff574513c63a"
I20240502 01:52:35.983093 21890 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7b5a257cabd146be941293466a8fbed1 in current term 2: Already voted for candidate 7f6298ac171e46209830ff574513c63a in this term.
W20240502 01:52:35.983741 21866 leader_election.cc:334] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7fdf79bd9d594be1942fbeef1d6d2c8f (127.20.65.124:38337): Network error: Client connection negotiation failed: client connection to 127.20.65.124:38337: connect: Connection refused (error 111)
I20240502 01:52:35.983894 21866 leader_election.cc:302] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f6298ac171e46209830ff574513c63a; no voters: 7b5a257cabd146be941293466a8fbed1, 7fdf79bd9d594be1942fbeef1d6d2c8f
W20240502 01:52:35.983990 21800 leader_election.cc:334] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7fdf79bd9d594be1942fbeef1d6d2c8f (127.20.65.124:38337): Network error: Client connection negotiation failed: client connection to 127.20.65.124:38337: connect: Connection refused (error 111)
I20240502 01:52:35.984153 21800 leader_election.cc:302] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b5a257cabd146be941293466a8fbed1; no voters: 7f6298ac171e46209830ff574513c63a, 7fdf79bd9d594be1942fbeef1d6d2c8f
I20240502 01:52:35.984395 22329 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 01:52:35.984464 22330 leader_election.cc:288] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [CANDIDATE]: Term 2 election: Requested vote from peers 7b5a257cabd146be941293466a8fbed1 (127.20.65.126:36605), 7fdf79bd9d594be1942fbeef1d6d2c8f (127.20.65.124:38337)
I20240502 01:52:35.984505 22335 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 01:52:37.823796 22342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:37.824018 22342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } }
I20240502 01:52:37.824764 22342 leader_election.cc:288] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7b5a257cabd146be941293466a8fbed1 (127.20.65.126:36605), 7fdf79bd9d594be1942fbeef1d6d2c8f (127.20.65.124:38337)
I20240502 01:52:37.825274 21824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f6298ac171e46209830ff574513c63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "7b5a257cabd146be941293466a8fbed1" is_pre_election: true
I20240502 01:52:37.825526 21824 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f6298ac171e46209830ff574513c63a in term 2.
W20240502 01:52:37.826011 21866 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.65.124:38337: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 01:52:37.826017 21864 leader_election.cc:302] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b5a257cabd146be941293466a8fbed1, 7f6298ac171e46209830ff574513c63a; no voters:
I20240502 01:52:37.826339 22342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 01:52:37.826543 22342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:37.826669 22342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 2 FOLLOWER]: Advancing to term 3
W20240502 01:52:37.827348 21866 leader_election.cc:334] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7fdf79bd9d594be1942fbeef1d6d2c8f (127.20.65.124:38337): Network error: Client connection negotiation failed: client connection to 127.20.65.124:38337: connect: Connection refused (error 111)
I20240502 01:52:37.877751 22342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } }
I20240502 01:52:37.878880 22342 leader_election.cc:288] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [CANDIDATE]: Term 3 election: Requested vote from peers 7b5a257cabd146be941293466a8fbed1 (127.20.65.126:36605), 7fdf79bd9d594be1942fbeef1d6d2c8f (127.20.65.124:38337)
I20240502 01:52:37.879123 21824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f6298ac171e46209830ff574513c63a" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "7b5a257cabd146be941293466a8fbed1"
I20240502 01:52:37.879371 21824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 2 FOLLOWER]: Advancing to term 3
W20240502 01:52:37.880156 21866 leader_election.cc:334] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7fdf79bd9d594be1942fbeef1d6d2c8f (127.20.65.124:38337): Network error: Client connection negotiation failed: client connection to 127.20.65.124:38337: connect: Connection refused (error 111)
I20240502 01:52:37.882583 21824 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f6298ac171e46209830ff574513c63a in term 3.
I20240502 01:52:37.883033 21864 leader_election.cc:302] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b5a257cabd146be941293466a8fbed1, 7f6298ac171e46209830ff574513c63a; no voters: 7fdf79bd9d594be1942fbeef1d6d2c8f
I20240502 01:52:37.883324 22342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:52:37.883776 22342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 3 LEADER]: Becoming Leader. State: Replica: 7f6298ac171e46209830ff574513c63a, State: Running, Role: LEADER
I20240502 01:52:37.884057 22342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } }
I20240502 01:52:37.885365 22345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f6298ac171e46209830ff574513c63a. Latest consensus state: current_term: 3 leader_uuid: "7f6298ac171e46209830ff574513c63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } } }
I20240502 01:52:37.885648 22345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:37.886140 22347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:37.888025 22347 catalog_manager.cc:669] Loaded metadata for table test-workload [id=9cf115c385ee431484aaa097921b39ba]
I20240502 01:52:37.889393 22347 tablet_loader.cc:96] loaded metadata for tablet 2b79da6d94c54f99ae9b3f0b066e4825 (table test-workload [id=9cf115c385ee431484aaa097921b39ba])
I20240502 01:52:37.889757 22347 tablet_loader.cc:96] loaded metadata for tablet a11207a1191b4774a90ad5291de4368c (table test-workload [id=9cf115c385ee431484aaa097921b39ba])
I20240502 01:52:37.890064 22347 tablet_loader.cc:96] loaded metadata for tablet ccbde4d521a5459f8d1433cc421324c5 (table test-workload [id=9cf115c385ee431484aaa097921b39ba])
I20240502 01:52:37.890285 22347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:37.891276 22347 catalog_manager.cc:1259] Loaded cluster ID: ba1241031aeb4ec590fda072ecb5c87e
I20240502 01:52:37.891363 22347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:37.892591 22347 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:37.893450 22347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a: Loaded TSK: 0
I20240502 01:52:37.894112 22347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:38.320894 22024 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:52:38.321062 22024 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:52:38.463915 21824 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 3 FOLLOWER]: Refusing update from remote peer 7f6298ac171e46209830ff574513c63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 01:52:38.464512 22349 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:52:38.478356 22351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f6298ac171e46209830ff574513c63a. Latest consensus state: current_term: 3 leader_uuid: "7f6298ac171e46209830ff574513c63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } } }
I20240502 01:52:38.478825 22351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:52:38.480908 22354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "7f6298ac171e46209830ff574513c63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } } }
I20240502 01:52:38.481292 22354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:38.484975 22351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "7f6298ac171e46209830ff574513c63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a257cabd146be941293466a8fbed1" member_type: VOTER last_known_addr { host: "127.20.65.126" port: 36605 } } peers { permanent_uuid: "7f6298ac171e46209830ff574513c63a" member_type: VOTER last_known_addr { host: "127.20.65.125" port: 39035 } } peers { permanent_uuid: "7fdf79bd9d594be1942fbeef1d6d2c8f" member_type: VOTER last_known_addr { host: "127.20.65.124" port: 38337 } } }
I20240502 01:52:38.485256 22351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [sys.catalog]: This master's current role is: FOLLOWER
W20240502 01:52:38.486891 21866 consensus_peers.cc:485] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a -> Peer 7fdf79bd9d594be1942fbeef1d6d2c8f (127.20.65.124:38337): Couldn't send request to peer 7fdf79bd9d594be1942fbeef1d6d2c8f. Status: Network error: Client connection negotiation failed: client connection to 127.20.65.124:38337: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:38.494882 20741 tablet_server.cc:178] TabletServer@127.20.65.65:0 shutting down...
I20240502 01:52:38.502780 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:52:38.503023 20741 tablet_replica.cc:330] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1: stopping tablet replica
I20240502 01:52:38.503203 20741 raft_consensus.cc:2229] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:52:38.503377 20741 raft_consensus.cc:2258] T ccbde4d521a5459f8d1433cc421324c5 P 8342461f3c1d4930a93e5eac1971b6f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:38.516893 20741 tablet_server.cc:194] TabletServer@127.20.65.65:0 shutdown complete.
I20240502 01:52:38.519901 20741 tablet_server.cc:178] TabletServer@127.20.65.66:0 shutting down...
I20240502 01:52:38.528901 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:52:38.529150 20741 tablet_replica.cc:330] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1: stopping tablet replica
I20240502 01:52:38.529348 20741 raft_consensus.cc:2229] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:52:38.529542 20741 raft_consensus.cc:2258] T 2b79da6d94c54f99ae9b3f0b066e4825 P 80ad8708a5cd4a80bd711523383e7ea1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:38.543247 20741 tablet_server.cc:194] TabletServer@127.20.65.66:0 shutdown complete.
I20240502 01:52:38.546018 20741 tablet_server.cc:178] TabletServer@127.20.65.67:0 shutting down...
I20240502 01:52:38.554257 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:52:38.554526 20741 tablet_replica.cc:330] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa: stopping tablet replica
I20240502 01:52:38.554721 20741 raft_consensus.cc:2229] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:52:38.554903 20741 raft_consensus.cc:2258] T a11207a1191b4774a90ad5291de4368c P 2ca785ff912d4397a4dc6eec2c9ce7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:38.569027 20741 tablet_server.cc:194] TabletServer@127.20.65.67:0 shutdown complete.
I20240502 01:52:38.571879 20741 master.cc:537] Master@127.20.65.126:36605 shutting down...
I20240502 01:52:38.579481 20741 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:38.579715 20741 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:38.579849 20741 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7b5a257cabd146be941293466a8fbed1: stopping tablet replica
I20240502 01:52:38.583679 20741 master.cc:552] Master@127.20.65.126:36605 shutdown complete.
I20240502 01:52:38.587831 20741 master.cc:537] Master@127.20.65.125:39035 shutting down...
I20240502 01:52:38.595283 20741 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 3 LEADER]: Raft consensus shutting down.
I20240502 01:52:38.595643 20741 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:38.595824 20741 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7f6298ac171e46209830ff574513c63a: stopping tablet replica
I20240502 01:52:38.599645 20741 master.cc:552] Master@127.20.65.125:39035 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6501 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240502 01:52:38.612128 20741 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.65.126:42707
I20240502 01:52:38.612633 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:38.615875 22362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.616652 22363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.617257 22365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:38.618212 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:38.618520 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:38.618583 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:38.618666 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614758618621 us; error 0 us; skew 500 ppm
I20240502 01:52:38.618839 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:38.625684 20741 webserver.cc:460] Webserver started at http://127.20.65.126:46853/ using document root <none> and password file <none>
I20240502 01:52:38.625913 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:38.626014 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:38.626155 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:38.626864 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/master-0-root/instance:
uuid: "14eb5b0ef2514056a0bfa58a7b49ef26"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-7cj2"
I20240502 01:52:38.629298 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 01:52:38.630970 22370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.631382 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240502 01:52:38.631487 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/master-0-root
uuid: "14eb5b0ef2514056a0bfa58a7b49ef26"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-7cj2"
I20240502 01:52:38.631621 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:38.651271 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:38.660779 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:38.672950 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.126:42707
I20240502 01:52:38.672987 22421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.126:42707 every 8 connection(s)
I20240502 01:52:38.674449 22422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:38.678303 22422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26: Bootstrap starting.
I20240502 01:52:38.679509 22422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:38.680917 22422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26: No bootstrap required, opened a new log
I20240502 01:52:38.681404 22422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eb5b0ef2514056a0bfa58a7b49ef26" member_type: VOTER }
I20240502 01:52:38.681520 22422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:38.681557 22422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14eb5b0ef2514056a0bfa58a7b49ef26, State: Initialized, Role: FOLLOWER
I20240502 01:52:38.681725 22422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eb5b0ef2514056a0bfa58a7b49ef26" member_type: VOTER }
I20240502 01:52:38.681840 22422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:38.681881 22422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:38.681931 22422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:38.769454 22422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eb5b0ef2514056a0bfa58a7b49ef26" member_type: VOTER }
I20240502 01:52:38.769804 22422 leader_election.cc:302] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14eb5b0ef2514056a0bfa58a7b49ef26; no voters:
I20240502 01:52:38.770318 22422 leader_election.cc:288] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:38.770505 22425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:38.771270 22425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [term 1 LEADER]: Becoming Leader. State: Replica: 14eb5b0ef2514056a0bfa58a7b49ef26, State: Running, Role: LEADER
I20240502 01:52:38.771498 22422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:38.771564 22425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eb5b0ef2514056a0bfa58a7b49ef26" member_type: VOTER }
I20240502 01:52:38.774871 22426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14eb5b0ef2514056a0bfa58a7b49ef26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eb5b0ef2514056a0bfa58a7b49ef26" member_type: VOTER } }
I20240502 01:52:38.775024 22426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:38.775372 22427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14eb5b0ef2514056a0bfa58a7b49ef26. Latest consensus state: current_term: 1 leader_uuid: "14eb5b0ef2514056a0bfa58a7b49ef26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eb5b0ef2514056a0bfa58a7b49ef26" member_type: VOTER } }
I20240502 01:52:38.775492 22427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:38.779546 22430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:38.781534 22430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:38.782156 20741 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:52:38.784416 22430 catalog_manager.cc:1347] Generated new cluster ID: c911c171d4c649719d7b59d11aa4299f
I20240502 01:52:38.784495 22430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:38.798290 22430 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:38.799059 22430 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:38.803977 22430 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26: Generated new TSK 0
I20240502 01:52:38.804168 22430 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:38.814719 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:38.817701 22444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.818675 22445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.819304 22447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:38.819986 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:38.820245 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:38.820307 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:38.820364 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614758820360 us; error 0 us; skew 500 ppm
I20240502 01:52:38.820520 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:38.822139 20741 webserver.cc:460] Webserver started at http://127.20.65.65:46033/ using document root <none> and password file <none>
I20240502 01:52:38.822364 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:38.822456 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:38.822604 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:38.823239 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-0-root/instance:
uuid: "bb616c04d0624a6fb7bb951fc5c716b3"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-7cj2"
I20240502 01:52:38.825699 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:52:38.827241 22452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.827594 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:52:38.827713 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-0-root
uuid: "bb616c04d0624a6fb7bb951fc5c716b3"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-7cj2"
I20240502 01:52:38.827816 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:38.837082 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:38.837772 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:38.838281 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:38.839109 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:38.839172 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.839252 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:38.839296 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.852793 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.65:44433
I20240502 01:52:38.852821 22514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.65:44433 every 8 connection(s)
I20240502 01:52:38.854804 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:38.858425 22519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.859027 22520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.859635 22522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:38.860069 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:38.860318 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:38.860368 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:38.860388 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614758860385 us; error 0 us; skew 500 ppm
I20240502 01:52:38.860471 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:38.861663 22515 heartbeater.cc:347] Connected to a master server at 127.20.65.126:42707
I20240502 01:52:38.861805 22515 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:38.861863 20741 webserver.cc:460] Webserver started at http://127.20.65.66:38633/ using document root <none> and password file <none>
I20240502 01:52:38.862035 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:38.862083 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:38.862159 22515 heartbeater.cc:511] Master 127.20.65.126:42707 requested a full tablet report, sending...
I20240502 01:52:38.862191 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:38.863094 22387 ts_manager.cc:194] Registered new tserver with Master: bb616c04d0624a6fb7bb951fc5c716b3 (127.20.65.65:44433)
I20240502 01:52:38.863108 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-1-root/instance:
uuid: "e43fa8934713488a985235fa31489622"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-7cj2"
I20240502 01:52:38.864109 22387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44038
I20240502 01:52:38.865633 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:38.867193 22527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.867563 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:52:38.867668 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-1-root
uuid: "e43fa8934713488a985235fa31489622"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-7cj2"
I20240502 01:52:38.867766 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:38.892608 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:38.893193 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:38.893762 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:38.894650 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:38.894708 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.894793 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:38.894829 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.910073 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.66:34141
I20240502 01:52:38.910109 22590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.66:34141 every 8 connection(s)
I20240502 01:52:38.911648 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:38.915741 22594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.916779 22595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.917126 22597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:38.917917 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:38.918211 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:38.918256 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:38.918278 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614758918275 us; error 0 us; skew 500 ppm
I20240502 01:52:38.918421 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:38.918712 22591 heartbeater.cc:347] Connected to a master server at 127.20.65.126:42707
I20240502 01:52:38.918845 22591 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:38.919174 22591 heartbeater.cc:511] Master 127.20.65.126:42707 requested a full tablet report, sending...
I20240502 01:52:38.919905 20741 webserver.cc:460] Webserver started at http://127.20.65.67:39853/ using document root <none> and password file <none>
I20240502 01:52:38.920015 22387 ts_manager.cc:194] Registered new tserver with Master: e43fa8934713488a985235fa31489622 (127.20.65.66:34141)
I20240502 01:52:38.920133 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:38.920202 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:38.920336 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:38.920785 22387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44048
I20240502 01:52:38.920990 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-2-root/instance:
uuid: "3a598e844c314fd7b0c661bffacb10e7"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-7cj2"
I20240502 01:52:38.923400 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:52:38.924988 22602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.925307 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:52:38.925402 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-2-root
uuid: "3a598e844c314fd7b0c661bffacb10e7"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-7cj2"
I20240502 01:52:38.925511 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:38.954262 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:38.954855 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:38.955439 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:38.956238 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:38.956301 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.956372 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:38.956410 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.970695 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.67:42439
I20240502 01:52:38.970726 22665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.67:42439 every 8 connection(s)
I20240502 01:52:38.977341 22666 heartbeater.cc:347] Connected to a master server at 127.20.65.126:42707
I20240502 01:52:38.977496 22666 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:38.977811 22666 heartbeater.cc:511] Master 127.20.65.126:42707 requested a full tablet report, sending...
I20240502 01:52:38.978629 22387 ts_manager.cc:194] Registered new tserver with Master: 3a598e844c314fd7b0c661bffacb10e7 (127.20.65.67:42439)
I20240502 01:52:38.979314 22387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44054
I20240502 01:52:38.979709 20741 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007912555s
I20240502 01:52:39.865353 22515 heartbeater.cc:503] Master 127.20.65.126:42707 was elected leader, sending a full tablet report...
I20240502 01:52:39.921957 22591 heartbeater.cc:503] Master 127.20.65.126:42707 was elected leader, sending a full tablet report...
I20240502 01:52:39.980799 22666 heartbeater.cc:503] Master 127.20.65.126:42707 was elected leader, sending a full tablet report...
I20240502 01:52:40.007536 20741 test_util.cc:268] Using random seed: -609065113
I20240502 01:52:40.020576 22387 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44064:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:52:40.021205 22387 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:52:40.042071 22480 tablet_service.cc:1467] Processing CreateTablet for tablet 7d24cf9c6ad349a3801c6859fa340af1 (DEFAULT_TABLE table=test-workload [id=53b7fb7158a74bb09490b160b93895df]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 01:52:40.042603 22480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d24cf9c6ad349a3801c6859fa340af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.046788 22479 tablet_service.cc:1467] Processing CreateTablet for tablet d9fce3e4207d4e988c54670aacb46c1f (DEFAULT_TABLE table=test-workload [id=53b7fb7158a74bb09490b160b93895df]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 01:52:40.047204 22479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9fce3e4207d4e988c54670aacb46c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.058142 22686 tablet_bootstrap.cc:492] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3: Bootstrap starting.
I20240502 01:52:40.061411 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 7d24cf9c6ad349a3801c6859fa340af1 (DEFAULT_TABLE table=test-workload [id=53b7fb7158a74bb09490b160b93895df]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 01:52:40.061501 22555 tablet_service.cc:1467] Processing CreateTablet for tablet d9fce3e4207d4e988c54670aacb46c1f (DEFAULT_TABLE table=test-workload [id=53b7fb7158a74bb09490b160b93895df]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 01:52:40.061722 22686 tablet_bootstrap.cc:654] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.061934 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d24cf9c6ad349a3801c6859fa340af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.066426 22686 tablet_bootstrap.cc:492] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3: No bootstrap required, opened a new log
I20240502 01:52:40.066493 22555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9fce3e4207d4e988c54670aacb46c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.066836 22631 tablet_service.cc:1467] Processing CreateTablet for tablet 7d24cf9c6ad349a3801c6859fa340af1 (DEFAULT_TABLE table=test-workload [id=53b7fb7158a74bb09490b160b93895df]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 01:52:40.066879 22630 tablet_service.cc:1467] Processing CreateTablet for tablet d9fce3e4207d4e988c54670aacb46c1f (DEFAULT_TABLE table=test-workload [id=53b7fb7158a74bb09490b160b93895df]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 01:52:40.067241 22686 ts_tablet_manager.cc:1367] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20240502 01:52:40.067297 22631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d24cf9c6ad349a3801c6859fa340af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.068145 22686 raft_consensus.cc:348] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } }
I20240502 01:52:40.068360 22686 raft_consensus.cc:374] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.068418 22686 raft_consensus.cc:729] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb616c04d0624a6fb7bb951fc5c716b3, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.068632 22686 consensus_queue.cc:260] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } }
I20240502 01:52:40.072002 22630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9fce3e4207d4e988c54670aacb46c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.079080 22688 tablet_bootstrap.cc:492] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622: Bootstrap starting.
I20240502 01:52:40.079658 22686 ts_tablet_manager.cc:1398] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20240502 01:52:40.079977 22686 tablet_bootstrap.cc:492] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3: Bootstrap starting.
I20240502 01:52:40.081318 22688 tablet_bootstrap.cc:654] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.081568 22686 tablet_bootstrap.cc:654] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.081918 22689 raft_consensus.cc:482] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:40.082109 22689 raft_consensus.cc:504] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } }
I20240502 01:52:40.090337 22689 leader_election.cc:288] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e43fa8934713488a985235fa31489622 (127.20.65.66:34141), 3a598e844c314fd7b0c661bffacb10e7 (127.20.65.67:42439)
I20240502 01:52:40.093760 22566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fce3e4207d4e988c54670aacb46c1f" candidate_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e43fa8934713488a985235fa31489622" is_pre_election: true
I20240502 01:52:40.093940 22690 tablet_bootstrap.cc:492] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7: Bootstrap starting.
W20240502 01:52:40.094797 22453 leader_election.cc:341] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e43fa8934713488a985235fa31489622 (127.20.65.66:34141): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:40.095796 22690 tablet_bootstrap.cc:654] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.097941 22686 tablet_bootstrap.cc:492] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3: No bootstrap required, opened a new log
I20240502 01:52:40.098265 22688 tablet_bootstrap.cc:492] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622: No bootstrap required, opened a new log
I20240502 01:52:40.098510 22688 ts_tablet_manager.cc:1367] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.013s
I20240502 01:52:40.100481 22686 ts_tablet_manager.cc:1367] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.006s
I20240502 01:52:40.100455 22688 raft_consensus.cc:348] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } }
I20240502 01:52:40.100718 22688 raft_consensus.cc:374] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.101076 22688 raft_consensus.cc:729] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e43fa8934713488a985235fa31489622, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.101519 22688 consensus_queue.cc:260] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } }
I20240502 01:52:40.102730 22686 raft_consensus.cc:348] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } }
I20240502 01:52:40.102818 22641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fce3e4207d4e988c54670aacb46c1f" candidate_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a598e844c314fd7b0c661bffacb10e7" is_pre_election: true
W20240502 01:52:40.103610 22455 leader_election.cc:341] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a598e844c314fd7b0c661bffacb10e7 (127.20.65.67:42439): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:40.104198 22688 ts_tablet_manager.cc:1398] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:52:40.104144 22455 leader_election.cc:302] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb616c04d0624a6fb7bb951fc5c716b3; no voters: 3a598e844c314fd7b0c661bffacb10e7, e43fa8934713488a985235fa31489622
I20240502 01:52:40.104441 22689 raft_consensus.cc:2726] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:52:40.104524 22690 tablet_bootstrap.cc:492] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7: No bootstrap required, opened a new log
I20240502 01:52:40.104149 22686 raft_consensus.cc:374] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.104719 22686 raft_consensus.cc:729] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb616c04d0624a6fb7bb951fc5c716b3, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.104526 22688 tablet_bootstrap.cc:492] T d9fce3e4207d4e988c54670aacb46c1f P e43fa8934713488a985235fa31489622: Bootstrap starting.
I20240502 01:52:40.104976 22686 consensus_queue.cc:260] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } }
I20240502 01:52:40.105471 22686 ts_tablet_manager.cc:1398] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:52:40.106593 22688 tablet_bootstrap.cc:654] T d9fce3e4207d4e988c54670aacb46c1f P e43fa8934713488a985235fa31489622: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.104749 22690 ts_tablet_manager.cc:1367] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20240502 01:52:40.108305 22690 raft_consensus.cc:348] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } }
I20240502 01:52:40.108587 22690 raft_consensus.cc:374] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.108644 22688 tablet_bootstrap.cc:492] T d9fce3e4207d4e988c54670aacb46c1f P e43fa8934713488a985235fa31489622: No bootstrap required, opened a new log
I20240502 01:52:40.108709 22690 raft_consensus.cc:729] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a598e844c314fd7b0c661bffacb10e7, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.108871 22688 ts_tablet_manager.cc:1367] T d9fce3e4207d4e988c54670aacb46c1f P e43fa8934713488a985235fa31489622: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240502 01:52:40.108951 22690 consensus_queue.cc:260] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } }
I20240502 01:52:40.109753 22690 ts_tablet_manager.cc:1398] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:52:40.110029 22690 tablet_bootstrap.cc:492] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7: Bootstrap starting.
I20240502 01:52:40.110236 22688 raft_consensus.cc:348] T d9fce3e4207d4e988c54670aacb46c1f P e43fa8934713488a985235fa31489622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } }
I20240502 01:52:40.110494 22688 raft_consensus.cc:374] T d9fce3e4207d4e988c54670aacb46c1f P e43fa8934713488a985235fa31489622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.110596 22688 raft_consensus.cc:729] T d9fce3e4207d4e988c54670aacb46c1f P e43fa8934713488a985235fa31489622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e43fa8934713488a985235fa31489622, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.110862 22688 consensus_queue.cc:260] T d9fce3e4207d4e988c54670aacb46c1f P e43fa8934713488a985235fa31489622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } }
I20240502 01:52:40.111629 22688 ts_tablet_manager.cc:1398] T d9fce3e4207d4e988c54670aacb46c1f P e43fa8934713488a985235fa31489622: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:52:40.111668 22690 tablet_bootstrap.cc:654] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.113440 22690 tablet_bootstrap.cc:492] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7: No bootstrap required, opened a new log
I20240502 01:52:40.113608 22690 ts_tablet_manager.cc:1367] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:40.114274 22690 raft_consensus.cc:348] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } }
I20240502 01:52:40.114547 22690 raft_consensus.cc:374] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.114632 22690 raft_consensus.cc:729] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a598e844c314fd7b0c661bffacb10e7, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.114840 22690 consensus_queue.cc:260] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } }
I20240502 01:52:40.115223 22690 ts_tablet_manager.cc:1398] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:40.253737 22698 raft_consensus.cc:482] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:40.253940 22698 raft_consensus.cc:504] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } }
I20240502 01:52:40.254832 22698 leader_election.cc:288] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb616c04d0624a6fb7bb951fc5c716b3 (127.20.65.65:44433), e43fa8934713488a985235fa31489622 (127.20.65.66:34141)
I20240502 01:52:40.261811 22490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fce3e4207d4e988c54670aacb46c1f" candidate_uuid: "3a598e844c314fd7b0c661bffacb10e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" is_pre_election: true
I20240502 01:52:40.262127 22490 raft_consensus.cc:2454] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a598e844c314fd7b0c661bffacb10e7 in term 0.
I20240502 01:52:40.262482 22566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fce3e4207d4e988c54670aacb46c1f" candidate_uuid: "3a598e844c314fd7b0c661bffacb10e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e43fa8934713488a985235fa31489622" is_pre_election: true
I20240502 01:52:40.262643 22604 leader_election.cc:302] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a598e844c314fd7b0c661bffacb10e7, bb616c04d0624a6fb7bb951fc5c716b3; no voters:
I20240502 01:52:40.262727 22566 raft_consensus.cc:2454] T d9fce3e4207d4e988c54670aacb46c1f P e43fa8934713488a985235fa31489622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a598e844c314fd7b0c661bffacb10e7 in term 0.
I20240502 01:52:40.262967 22698 raft_consensus.cc:2781] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:40.263096 22698 raft_consensus.cc:482] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:40.263200 22698 raft_consensus.cc:3037] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:40.308805 22689 raft_consensus.cc:482] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:40.309007 22689 raft_consensus.cc:504] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } }
I20240502 01:52:40.309685 22689 leader_election.cc:288] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a598e844c314fd7b0c661bffacb10e7 (127.20.65.67:42439), e43fa8934713488a985235fa31489622 (127.20.65.66:34141)
I20240502 01:52:40.310201 22566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d24cf9c6ad349a3801c6859fa340af1" candidate_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e43fa8934713488a985235fa31489622" is_pre_election: true
I20240502 01:52:40.310206 22641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d24cf9c6ad349a3801c6859fa340af1" candidate_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a598e844c314fd7b0c661bffacb10e7" is_pre_election: true
I20240502 01:52:40.310484 22566 raft_consensus.cc:2454] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb616c04d0624a6fb7bb951fc5c716b3 in term 0.
I20240502 01:52:40.310505 22641 raft_consensus.cc:2454] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb616c04d0624a6fb7bb951fc5c716b3 in term 0.
I20240502 01:52:40.310899 22453 leader_election.cc:302] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb616c04d0624a6fb7bb951fc5c716b3, e43fa8934713488a985235fa31489622; no voters:
I20240502 01:52:40.311177 22689 raft_consensus.cc:2781] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:40.311273 22689 raft_consensus.cc:482] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:40.311336 22689 raft_consensus.cc:3037] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:40.358198 22698 raft_consensus.cc:504] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } }
I20240502 01:52:40.358271 22689 raft_consensus.cc:504] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } }
I20240502 01:52:40.358999 22698 leader_election.cc:288] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [CANDIDATE]: Term 1 election: Requested vote from peers bb616c04d0624a6fb7bb951fc5c716b3 (127.20.65.65:44433), e43fa8934713488a985235fa31489622 (127.20.65.66:34141)
I20240502 01:52:40.359491 22490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fce3e4207d4e988c54670aacb46c1f" candidate_uuid: "3a598e844c314fd7b0c661bffacb10e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb616c04d0624a6fb7bb951fc5c716b3"
I20240502 01:52:40.359519 22566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d24cf9c6ad349a3801c6859fa340af1" candidate_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e43fa8934713488a985235fa31489622"
I20240502 01:52:40.359686 22490 raft_consensus.cc:3037] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:40.359730 22566 raft_consensus.cc:3037] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:40.359787 22689 leader_election.cc:288] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [CANDIDATE]: Term 1 election: Requested vote from peers 3a598e844c314fd7b0c661bffacb10e7 (127.20.65.67:42439), e43fa8934713488a985235fa31489622 (127.20.65.66:34141)
I20240502 01:52:40.360533 22565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fce3e4207d4e988c54670aacb46c1f" candidate_uuid: "3a598e844c314fd7b0c661bffacb10e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e43fa8934713488a985235fa31489622"
I20240502 01:52:40.360714 22565 raft_consensus.cc:3037] T d9fce3e4207d4e988c54670aacb46c1f P e43fa8934713488a985235fa31489622 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:40.362723 22490 raft_consensus.cc:2454] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a598e844c314fd7b0c661bffacb10e7 in term 1.
I20240502 01:52:40.363058 22566 raft_consensus.cc:2454] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb616c04d0624a6fb7bb951fc5c716b3 in term 1.
I20240502 01:52:40.363166 22604 leader_election.cc:302] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a598e844c314fd7b0c661bffacb10e7, bb616c04d0624a6fb7bb951fc5c716b3; no voters:
I20240502 01:52:40.363521 22453 leader_election.cc:302] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb616c04d0624a6fb7bb951fc5c716b3, e43fa8934713488a985235fa31489622; no voters:
I20240502 01:52:40.363523 22698 raft_consensus.cc:2781] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:40.363731 22641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d24cf9c6ad349a3801c6859fa340af1" candidate_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a598e844c314fd7b0c661bffacb10e7"
I20240502 01:52:40.363868 22689 raft_consensus.cc:2781] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:40.363963 22641 raft_consensus.cc:3037] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:40.364109 22698 raft_consensus.cc:686] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [term 1 LEADER]: Becoming Leader. State: Replica: 3a598e844c314fd7b0c661bffacb10e7, State: Running, Role: LEADER
I20240502 01:52:40.364444 22698 consensus_queue.cc:237] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } }
I20240502 01:52:40.364897 22689 raft_consensus.cc:686] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [term 1 LEADER]: Becoming Leader. State: Replica: bb616c04d0624a6fb7bb951fc5c716b3, State: Running, Role: LEADER
I20240502 01:52:40.365424 22689 consensus_queue.cc:237] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } }
I20240502 01:52:40.367388 22386 catalog_manager.cc:5484] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a598e844c314fd7b0c661bffacb10e7 (127.20.65.67). New cstate: current_term: 1 leader_uuid: "3a598e844c314fd7b0c661bffacb10e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:40.368224 22385 catalog_manager.cc:5484] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb616c04d0624a6fb7bb951fc5c716b3 (127.20.65.65). New cstate: current_term: 1 leader_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:40.460050 22565 raft_consensus.cc:2454] T d9fce3e4207d4e988c54670aacb46c1f P e43fa8934713488a985235fa31489622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a598e844c314fd7b0c661bffacb10e7 in term 1.
I20240502 01:52:40.460050 22641 raft_consensus.cc:2454] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb616c04d0624a6fb7bb951fc5c716b3 in term 1.
I20240502 01:52:40.782197 22441 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:52:40.782820 22441 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:52:40.782958 22441 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:52:40.801553 22698 consensus_queue.cc:1035] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:40.803740 22689 consensus_queue.cc:1035] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:40.807127 22698 consensus_queue.cc:1035] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:40.820895 22714 consensus_queue.cc:1035] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:42.783433 22441 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:52:42.784092 22441 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:52:42.784237 22441 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:52:44.784713 22441 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:52:44.785359 22441 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:52:44.785528 22441 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:52:46.785972 22441 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:52:46.786640 22441 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:52:46.786777 22441 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:52:48.787248 22441 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:52:48.787902 22441 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:52:48.788065 22441 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:52:50.788508 22441 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:52:50.789160 22441 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:52:50.789297 22441 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:52:51.451638 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:51.455574 22723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:51.456466 22724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:51.457156 22726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:51.458225 20741 server_base.cc:1034] running on GCE node
I20240502 01:52:51.458523 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:51.458602 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:51.458653 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614771458633 us; error 0 us; skew 500 ppm
I20240502 01:52:51.458784 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:51.466550 20741 webserver.cc:460] Webserver started at http://127.20.65.68:34007/ using document root <none> and password file <none>
I20240502 01:52:51.466826 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:51.466898 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:51.467020 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:51.467693 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-3-root/instance:
uuid: "ffe854c500b5420c998152d88065a2ec"
format_stamp: "Formatted at 2024-05-02 01:52:51 on dist-test-slave-7cj2"
I20240502 01:52:51.470481 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20240502 01:52:51.472205 22732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:51.472630 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:52:51.472733 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-3-root
uuid: "ffe854c500b5420c998152d88065a2ec"
format_stamp: "Formatted at 2024-05-02 01:52:51 on dist-test-slave-7cj2"
I20240502 01:52:51.472855 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:51.483768 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:51.484308 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:51.484869 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:51.485746 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:51.485826 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:51.485922 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:51.485996 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:51.501029 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.68:34661
I20240502 01:52:51.501084 22795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.68:34661 every 8 connection(s)
I20240502 01:52:51.507663 22796 heartbeater.cc:347] Connected to a master server at 127.20.65.126:42707
I20240502 01:52:51.507784 22796 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:51.508157 22796 heartbeater.cc:511] Master 127.20.65.126:42707 requested a full tablet report, sending...
I20240502 01:52:51.508996 22386 ts_manager.cc:194] Registered new tserver with Master: ffe854c500b5420c998152d88065a2ec (127.20.65.68:34661)
I20240502 01:52:51.509776 22386 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48866
I20240502 01:52:51.803125 22490 consensus_queue.cc:237] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } attrs { replace: true } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: true } }
I20240502 01:52:51.805982 22641 raft_consensus.cc:1261] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7 [term 1 FOLLOWER]: Refusing update from remote peer bb616c04d0624a6fb7bb951fc5c716b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:51.806030 22565 raft_consensus.cc:1261] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622 [term 1 FOLLOWER]: Refusing update from remote peer bb616c04d0624a6fb7bb951fc5c716b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:51.806592 22722 consensus_queue.cc:1035] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:51.806986 22803 consensus_queue.cc:1035] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:51.810037 22722 raft_consensus.cc:2932] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ffe854c500b5420c998152d88065a2ec (127.20.65.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } attrs { replace: true } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: true } } }
I20240502 01:52:51.810655 22641 raft_consensus.cc:2932] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ffe854c500b5420c998152d88065a2ec (127.20.65.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } attrs { replace: true } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: true } } }
I20240502 01:52:51.811779 22565 raft_consensus.cc:2932] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ffe854c500b5420c998152d88065a2ec (127.20.65.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } attrs { replace: true } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: true } } }
W20240502 01:52:51.813593 22455 consensus_peers.cc:485] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 -> Peer ffe854c500b5420c998152d88065a2ec (127.20.65.68:34661): Couldn't send request to peer ffe854c500b5420c998152d88065a2ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d24cf9c6ad349a3801c6859fa340af1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:51.815933 22386 catalog_manager.cc:5484] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7 reported cstate change: config changed from index -1 to 2, NON_VOTER ffe854c500b5420c998152d88065a2ec (127.20.65.68) added. New cstate: current_term: 1 leader_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } attrs { replace: true } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: true } } }
I20240502 01:52:52.313097 22814 ts_tablet_manager.cc:902] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec: Initiating tablet copy from peer bb616c04d0624a6fb7bb951fc5c716b3 (127.20.65.65:44433)
I20240502 01:52:52.314898 22814 tablet_copy_client.cc:308] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec: tablet copy: Beginning tablet copy session from remote peer at address 127.20.65.65:44433
I20240502 01:52:52.322543 22500 tablet_copy_service.cc:140] P bb616c04d0624a6fb7bb951fc5c716b3: Received BeginTabletCopySession request for tablet 7d24cf9c6ad349a3801c6859fa340af1 from peer ffe854c500b5420c998152d88065a2ec ({username='slave'} at 127.0.0.1:44934)
I20240502 01:52:52.322778 22500 tablet_copy_service.cc:161] P bb616c04d0624a6fb7bb951fc5c716b3: Beginning new tablet copy session on tablet 7d24cf9c6ad349a3801c6859fa340af1 from peer ffe854c500b5420c998152d88065a2ec at {username='slave'} at 127.0.0.1:44934: session id = ffe854c500b5420c998152d88065a2ec-7d24cf9c6ad349a3801c6859fa340af1
I20240502 01:52:52.326443 22500 tablet_copy_source_session.cc:205] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:52.328411 22814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d24cf9c6ad349a3801c6859fa340af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.334540 22814 tablet_copy_client.cc:783] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:52.334861 22814 tablet_copy_client.cc:647] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:52.336808 22814 tablet_copy_client.cc:523] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:52.340094 22814 tablet_bootstrap.cc:492] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec: Bootstrap starting.
I20240502 01:52:52.347108 22814 tablet_bootstrap.cc:492] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:52.347450 22814 tablet_bootstrap.cc:492] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec: Bootstrap complete.
I20240502 01:52:52.347712 22814 ts_tablet_manager.cc:1367] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20240502 01:52:52.348415 22814 raft_consensus.cc:348] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } attrs { replace: true } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: true } }
I20240502 01:52:52.348603 22814 raft_consensus.cc:729] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ffe854c500b5420c998152d88065a2ec, State: Initialized, Role: LEARNER
I20240502 01:52:52.348791 22814 consensus_queue.cc:260] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } attrs { replace: true } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: true } }
I20240502 01:52:52.349583 22796 heartbeater.cc:503] Master 127.20.65.126:42707 was elected leader, sending a full tablet report...
I20240502 01:52:52.349598 22814 ts_tablet_manager.cc:1398] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:52:52.350291 22500 tablet_copy_service.cc:342] P bb616c04d0624a6fb7bb951fc5c716b3: Request end of tablet copy session ffe854c500b5420c998152d88065a2ec-7d24cf9c6ad349a3801c6859fa340af1 received from {username='slave'} at 127.0.0.1:44934
I20240502 01:52:52.350463 22500 tablet_copy_service.cc:433] P bb616c04d0624a6fb7bb951fc5c716b3: ending tablet copy session ffe854c500b5420c998152d88065a2ec-7d24cf9c6ad349a3801c6859fa340af1 on tablet 7d24cf9c6ad349a3801c6859fa340af1 with peer ffe854c500b5420c998152d88065a2ec
I20240502 01:52:52.789690 22441 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:52:52.790642 22441 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:52:52.790809 22441 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:52:52.935424 22771 raft_consensus.cc:1203] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:52:53.405889 22824 raft_consensus.cc:1050] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3: attempting to promote NON_VOTER ffe854c500b5420c998152d88065a2ec to VOTER
I20240502 01:52:53.406723 22824 consensus_queue.cc:237] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } attrs { replace: true } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: false } }
I20240502 01:52:53.408851 22771 raft_consensus.cc:1261] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec [term 1 LEARNER]: Refusing update from remote peer bb616c04d0624a6fb7bb951fc5c716b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:52:53.408876 22565 raft_consensus.cc:1261] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622 [term 1 FOLLOWER]: Refusing update from remote peer bb616c04d0624a6fb7bb951fc5c716b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:52:53.409027 22641 raft_consensus.cc:1261] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7 [term 1 FOLLOWER]: Refusing update from remote peer bb616c04d0624a6fb7bb951fc5c716b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:52:53.409456 22827 consensus_queue.cc:1035] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:52:53.409778 22823 consensus_queue.cc:1035] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:52:53.410017 22722 consensus_queue.cc:1035] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:52:53.415717 22722 raft_consensus.cc:2932] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ffe854c500b5420c998152d88065a2ec (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } attrs { replace: true } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: false } } }
I20240502 01:52:53.416702 22641 raft_consensus.cc:2932] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ffe854c500b5420c998152d88065a2ec (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } attrs { replace: true } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: false } } }
I20240502 01:52:53.418784 22565 raft_consensus.cc:2932] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ffe854c500b5420c998152d88065a2ec (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } attrs { replace: true } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: false } } }
I20240502 01:52:53.420063 22771 raft_consensus.cc:2932] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ffe854c500b5420c998152d88065a2ec (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } attrs { replace: true } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: false } } }
I20240502 01:52:53.420292 22387 catalog_manager.cc:5484] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 reported cstate change: config changed from index 2 to 3, ffe854c500b5420c998152d88065a2ec (127.20.65.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a598e844c314fd7b0c661bffacb10e7" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 42439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:53.430123 22490 consensus_queue.cc:237] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: false } }
I20240502 01:52:53.431916 22565 raft_consensus.cc:1261] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622 [term 1 FOLLOWER]: Refusing update from remote peer bb616c04d0624a6fb7bb951fc5c716b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:52:53.432152 22771 raft_consensus.cc:1261] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec [term 1 FOLLOWER]: Refusing update from remote peer bb616c04d0624a6fb7bb951fc5c716b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:52:53.432441 22722 consensus_queue.cc:1035] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:52:53.432695 22824 consensus_queue.cc:1035] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:52:53.434753 22827 raft_consensus.cc:2932] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a598e844c314fd7b0c661bffacb10e7 (127.20.65.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: false } } }
I20240502 01:52:53.435420 22565 raft_consensus.cc:2932] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a598e844c314fd7b0c661bffacb10e7 (127.20.65.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: false } } }
I20240502 01:52:53.435388 22771 raft_consensus.cc:2932] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a598e844c314fd7b0c661bffacb10e7 (127.20.65.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: false } } }
I20240502 01:52:53.438259 22372 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d24cf9c6ad349a3801c6859fa340af1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:52:53.439595 22386 catalog_manager.cc:5484] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622 reported cstate change: config changed from index 3 to 4, VOTER 3a598e844c314fd7b0c661bffacb10e7 (127.20.65.67) evicted. New cstate: current_term: 1 leader_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb616c04d0624a6fb7bb951fc5c716b3" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 44433 } } peers { permanent_uuid: "e43fa8934713488a985235fa31489622" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 34141 } } peers { permanent_uuid: "ffe854c500b5420c998152d88065a2ec" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 34661 } attrs { promote: false } } }
I20240502 01:52:53.443235 22631 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d24cf9c6ad349a3801c6859fa340af1 with delete_type TABLET_DATA_TOMBSTONED (TS 3a598e844c314fd7b0c661bffacb10e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58730
I20240502 01:52:53.443946 22836 tablet_replica.cc:330] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7: stopping tablet replica
I20240502 01:52:53.444108 22836 raft_consensus.cc:2229] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:53.444331 22836 raft_consensus.cc:2258] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:53.445087 22836 ts_tablet_manager.cc:1875] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:52:53.449236 22836 ts_tablet_manager.cc:1888] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:52:53.449357 22836 log.cc:1196] T 7d24cf9c6ad349a3801c6859fa340af1 P 3a598e844c314fd7b0c661bffacb10e7: Deleting WAL directory at /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614741486931-20741-0/minicluster-data/ts-2-root/wals/7d24cf9c6ad349a3801c6859fa340af1
I20240502 01:52:53.449995 22373 catalog_manager.cc:4830] TS 3a598e844c314fd7b0c661bffacb10e7 (127.20.65.67:42439): tablet 7d24cf9c6ad349a3801c6859fa340af1 (table test-workload [id=53b7fb7158a74bb09490b160b93895df]) successfully deleted
I20240502 01:52:54.791253 22441 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:52:54.791885 22441 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:52:54.792002 22441 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:52:56.792392 22441 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:52:56.793037 22441 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:52:56.793140 22441 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:52:58.793515 22441 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:52:58.794111 22441 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:52:58.794241 22441 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:00.794620 22441 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:00.795217 22441 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:53:00.795325 22441 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:02.795739 22441 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:02.796378 22441 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:53:02.796490 22441 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:04.796981 22441 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:04.797672 22441 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:53:04.797804 22441 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:05.592878 20741 tablet_server.cc:178] TabletServer@127.20.65.65:0 shutting down...
I20240502 01:53:05.604048 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:53:05.604290 20741 tablet_replica.cc:330] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3: stopping tablet replica
I20240502 01:53:05.604502 20741 raft_consensus.cc:2229] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:05.604876 20741 raft_consensus.cc:2258] T 7d24cf9c6ad349a3801c6859fa340af1 P bb616c04d0624a6fb7bb951fc5c716b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:05.605527 20741 tablet_replica.cc:330] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3: stopping tablet replica
I20240502 01:53:05.605679 20741 raft_consensus.cc:2229] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:05.605840 20741 raft_consensus.cc:2258] T d9fce3e4207d4e988c54670aacb46c1f P bb616c04d0624a6fb7bb951fc5c716b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:05.625947 20741 tablet_server.cc:194] TabletServer@127.20.65.65:0 shutdown complete.
I20240502 01:53:05.630956 20741 tablet_server.cc:178] TabletServer@127.20.65.66:0 shutting down...
I20240502 01:53:05.643532 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:53:05.643774 20741 tablet_replica.cc:330] T d9fce3e4207d4e988c54670aacb46c1f P e43fa8934713488a985235fa31489622: stopping tablet replica
I20240502 01:53:05.643985 20741 raft_consensus.cc:2229] T d9fce3e4207d4e988c54670aacb46c1f P e43fa8934713488a985235fa31489622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:05.644155 20741 raft_consensus.cc:2258] T d9fce3e4207d4e988c54670aacb46c1f P e43fa8934713488a985235fa31489622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:05.644649 20741 tablet_replica.cc:330] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622: stopping tablet replica
I20240502 01:53:05.644791 20741 raft_consensus.cc:2229] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:05.645020 20741 raft_consensus.cc:2258] T 7d24cf9c6ad349a3801c6859fa340af1 P e43fa8934713488a985235fa31489622 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:53:05.648571 22604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.65.65:44433: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20240502 01:53:05.649567 20741 tablet_server.cc:194] TabletServer@127.20.65.66:0 shutdown complete.
W20240502 01:53:05.650368 22604 consensus_peers.cc:485] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 -> Peer bb616c04d0624a6fb7bb951fc5c716b3 (127.20.65.65:44433): Couldn't send request to peer bb616c04d0624a6fb7bb951fc5c716b3. Status: Network error: Client connection negotiation failed: client connection to 127.20.65.65:44433: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:05.652976 20741 tablet_server.cc:178] TabletServer@127.20.65.67:0 shutting down...
I20240502 01:53:05.663933 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:53:05.664146 20741 tablet_replica.cc:330] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7: stopping tablet replica
I20240502 01:53:05.664389 20741 raft_consensus.cc:2229] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:05.664675 20741 raft_consensus.cc:2258] T d9fce3e4207d4e988c54670aacb46c1f P 3a598e844c314fd7b0c661bffacb10e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:05.678618 20741 tablet_server.cc:194] TabletServer@127.20.65.67:0 shutdown complete.
I20240502 01:53:05.681849 20741 tablet_server.cc:178] TabletServer@127.20.65.68:0 shutting down...
I20240502 01:53:05.690547 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:53:05.690752 20741 tablet_replica.cc:330] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec: stopping tablet replica
I20240502 01:53:05.696492 20741 raft_consensus.cc:2229] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:05.696738 20741 raft_consensus.cc:2258] T 7d24cf9c6ad349a3801c6859fa340af1 P ffe854c500b5420c998152d88065a2ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:05.710894 20741 tablet_server.cc:194] TabletServer@127.20.65.68:0 shutdown complete.
I20240502 01:53:05.713594 20741 master.cc:537] Master@127.20.65.126:42707 shutting down...
I20240502 01:53:05.720994 20741 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:05.721194 20741 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:05.721285 20741 tablet_replica.cc:330] T 00000000000000000000000000000000 P 14eb5b0ef2514056a0bfa58a7b49ef26: stopping tablet replica
I20240502 01:53:05.734057 20741 master.cc:552] Master@127.20.65.126:42707 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (27133 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240502 01:53:05.745335 20741 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.65.126:34379
I20240502 01:53:05.745836 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:05.748139 22843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:05.748456 22844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:05.748858 22846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:05.748867 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:05.749183 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:05.749238 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:05.749266 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614785749263 us; error 0 us; skew 500 ppm
I20240502 01:53:05.749367 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:05.750546 20741 webserver.cc:460] Webserver started at http://127.20.65.126:45459/ using document root <none> and password file <none>
I20240502 01:53:05.750741 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:05.750789 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:05.750871 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:05.751433 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714614741486931-20741-0/minicluster-data/master-0-root/instance:
uuid: "c0757a41c60643d8b8f735cf2f02de20"
format_stamp: "Formatted at 2024-05-02 01:53:05 on dist-test-slave-7cj2"
I20240502 01:53:05.753739 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:05.755138 22851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:05.755461 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:53:05.755563 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714614741486931-20741-0/minicluster-data/master-0-root
uuid: "c0757a41c60643d8b8f735cf2f02de20"
format_stamp: "Formatted at 2024-05-02 01:53:05 on dist-test-slave-7cj2"
I20240502 01:53:05.755674 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714614741486931-20741-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714614741486931-20741-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714614741486931-20741-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:05.781342 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:05.781838 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:05.792276 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.126:34379
I20240502 01:53:05.792297 22903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.126:34379 every 8 connection(s)
I20240502 01:53:05.793450 22904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:05.797295 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20: Bootstrap starting.
I20240502 01:53:05.798310 22904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:05.799727 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20: No bootstrap required, opened a new log
I20240502 01:53:05.800261 22904 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0757a41c60643d8b8f735cf2f02de20" member_type: VOTER }
I20240502 01:53:05.800382 22904 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:05.800420 22904 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0757a41c60643d8b8f735cf2f02de20, State: Initialized, Role: FOLLOWER
I20240502 01:53:05.800582 22904 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0757a41c60643d8b8f735cf2f02de20" member_type: VOTER }
I20240502 01:53:05.800694 22904 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:53:05.800733 22904 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:53:05.800782 22904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:05.802367 22904 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0757a41c60643d8b8f735cf2f02de20" member_type: VOTER }
I20240502 01:53:05.802553 22904 leader_election.cc:302] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0757a41c60643d8b8f735cf2f02de20; no voters:
I20240502 01:53:05.802858 22904 leader_election.cc:288] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:53:05.802975 22907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:05.803409 22907 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [term 1 LEADER]: Becoming Leader. State: Replica: c0757a41c60643d8b8f735cf2f02de20, State: Running, Role: LEADER
I20240502 01:53:05.803651 22904 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:53:05.803612 22907 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0757a41c60643d8b8f735cf2f02de20" member_type: VOTER }
I20240502 01:53:05.804327 22909 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0757a41c60643d8b8f735cf2f02de20. Latest consensus state: current_term: 1 leader_uuid: "c0757a41c60643d8b8f735cf2f02de20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0757a41c60643d8b8f735cf2f02de20" member_type: VOTER } }
I20240502 01:53:05.804494 22909 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:05.804510 22908 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0757a41c60643d8b8f735cf2f02de20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0757a41c60643d8b8f735cf2f02de20" member_type: VOTER } }
I20240502 01:53:05.804623 22908 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:05.805665 22911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:53:05.807027 22911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:53:05.808959 20741 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:53:05.809849 22911 catalog_manager.cc:1347] Generated new cluster ID: b7012e1c62344853b24b22f231a49ba8
I20240502 01:53:05.809909 22911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:53:05.826519 22911 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:53:05.827216 22911 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:53:05.849678 22911 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20: Generated new TSK 0
I20240502 01:53:05.849876 22911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:53:05.873517 20741 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 7.7893e-05s
I20240502 01:53:17.956243 20741 master.cc:537] Master@127.20.65.126:34379 shutting down...
I20240502 01:53:17.970836 20741 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:17.971092 20741 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:17.971191 20741 tablet_replica.cc:330] T 00000000000000000000000000000000 P c0757a41c60643d8b8f735cf2f02de20: stopping tablet replica
I20240502 01:53:17.984934 20741 master.cc:552] Master@127.20.65.126:34379 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12248 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240502 01:53:17.993685 20741 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.65.126:35019
I20240502 01:53:17.994153 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:17.996899 22926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:17.997211 22927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:17.997864 22929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:17.999123 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:18.000905 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:18.000979 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:18.001040 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614798001021 us; error 0 us; skew 500 ppm
I20240502 01:53:18.001200 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:18.006408 20741 webserver.cc:460] Webserver started at http://127.20.65.126:35287/ using document root <none> and password file <none>
I20240502 01:53:18.006677 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:18.006755 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:18.006902 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:18.007529 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/master-0-root/instance:
uuid: "20f234f048d347e48c9501301d0dbc75"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-7cj2"
I20240502 01:53:18.009958 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:18.011477 22934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:18.011806 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:53:18.011919 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/master-0-root
uuid: "20f234f048d347e48c9501301d0dbc75"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-7cj2"
I20240502 01:53:18.012034 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:18.029660 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:18.030107 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:18.041085 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.126:35019
I20240502 01:53:18.041108 22985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.126:35019 every 8 connection(s)
I20240502 01:53:18.042315 22986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.046375 22986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75: Bootstrap starting.
I20240502 01:53:18.047454 22986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:18.048797 22986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75: No bootstrap required, opened a new log
I20240502 01:53:18.049360 22986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f234f048d347e48c9501301d0dbc75" member_type: VOTER }
I20240502 01:53:18.049490 22986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:18.049568 22986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f234f048d347e48c9501301d0dbc75, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.049785 22986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f234f048d347e48c9501301d0dbc75" member_type: VOTER }
I20240502 01:53:18.049919 22986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:53:18.050004 22986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:53:18.050096 22986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.051888 22986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f234f048d347e48c9501301d0dbc75" member_type: VOTER }
I20240502 01:53:18.052073 22986 leader_election.cc:302] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20f234f048d347e48c9501301d0dbc75; no voters:
I20240502 01:53:18.052455 22986 leader_election.cc:288] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:53:18.052579 22989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:18.053092 22989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [term 1 LEADER]: Becoming Leader. State: Replica: 20f234f048d347e48c9501301d0dbc75, State: Running, Role: LEADER
I20240502 01:53:18.053333 22986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:53:18.053323 22989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f234f048d347e48c9501301d0dbc75" member_type: VOTER }
I20240502 01:53:18.054313 22990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20f234f048d347e48c9501301d0dbc75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f234f048d347e48c9501301d0dbc75" member_type: VOTER } }
I20240502 01:53:18.054466 22991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20f234f048d347e48c9501301d0dbc75. Latest consensus state: current_term: 1 leader_uuid: "20f234f048d347e48c9501301d0dbc75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f234f048d347e48c9501301d0dbc75" member_type: VOTER } }
I20240502 01:53:18.054615 22990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:18.054677 22991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:18.058890 22994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:53:18.059000 20741 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:53:18.060071 22994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:53:18.062808 22994 catalog_manager.cc:1347] Generated new cluster ID: 2a0fd642e9584fc9a9e0541619ff9537
I20240502 01:53:18.062889 22994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:53:18.086917 22994 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:53:18.087671 22994 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:53:18.095459 22994 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75: Generated new TSK 0
I20240502 01:53:18.095696 22994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:53:18.123819 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:18.126718 23007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:18.127203 23008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:18.127732 23010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:18.128029 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:18.128252 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:18.128330 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:18.128401 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614798128371 us; error 0 us; skew 500 ppm
I20240502 01:53:18.128559 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:18.129798 20741 webserver.cc:460] Webserver started at http://127.20.65.65:45433/ using document root <none> and password file <none>
I20240502 01:53:18.130028 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:18.130113 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:18.130263 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:18.131011 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/ts-0-root/instance:
uuid: "10cdefbf6ad7486ca0f14ba98166c700"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-7cj2"
I20240502 01:53:18.133553 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 01:53:18.135239 23015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:18.135666 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:53:18.135782 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/ts-0-root
uuid: "10cdefbf6ad7486ca0f14ba98166c700"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-7cj2"
I20240502 01:53:18.135907 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:18.152256 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:18.152782 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:18.153403 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:18.154335 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:18.154453 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:18.154557 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:18.154628 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:18.167991 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.65:42605
I20240502 01:53:18.168016 23077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.65:42605 every 8 connection(s)
I20240502 01:53:18.169754 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:18.173182 23082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:18.174026 23085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:18.174350 23083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:18.175190 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:18.177731 23078 heartbeater.cc:347] Connected to a master server at 127.20.65.126:35019
I20240502 01:53:18.177870 23078 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:18.178220 23078 heartbeater.cc:511] Master 127.20.65.126:35019 requested a full tablet report, sending...
I20240502 01:53:18.179275 22951 ts_manager.cc:194] Registered new tserver with Master: 10cdefbf6ad7486ca0f14ba98166c700 (127.20.65.65:42605)
I20240502 01:53:18.180732 22951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33132
I20240502 01:53:18.185428 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:18.185492 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:18.185547 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614798185531 us; error 0 us; skew 500 ppm
I20240502 01:53:18.185648 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:18.186796 20741 webserver.cc:460] Webserver started at http://127.20.65.66:33261/ using document root <none> and password file <none>
I20240502 01:53:18.186962 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:18.187013 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:18.187093 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:18.187594 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/ts-1-root/instance:
uuid: "d339073cba9e4cf29879b84ad1715bfe"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-7cj2"
I20240502 01:53:18.189675 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:18.191035 23090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:18.191310 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240502 01:53:18.191390 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/ts-1-root
uuid: "d339073cba9e4cf29879b84ad1715bfe"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-7cj2"
I20240502 01:53:18.191486 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:18.206799 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:18.207228 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:18.207703 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:18.208438 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:18.208494 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:18.208559 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:18.208595 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:18.221028 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.66:37413
I20240502 01:53:18.221064 23152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.66:37413 every 8 connection(s)
I20240502 01:53:18.222664 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:18.225648 23156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:18.225819 23157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:18.227171 23159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:18.227403 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:18.227958 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:18.228041 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:18.228086 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614798228082 us; error 0 us; skew 500 ppm
I20240502 01:53:18.228219 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:18.228938 23153 heartbeater.cc:347] Connected to a master server at 127.20.65.126:35019
I20240502 01:53:18.229071 23153 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:18.229501 23153 heartbeater.cc:511] Master 127.20.65.126:35019 requested a full tablet report, sending...
I20240502 01:53:18.229565 20741 webserver.cc:460] Webserver started at http://127.20.65.67:33511/ using document root <none> and password file <none>
I20240502 01:53:18.229782 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:18.229864 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:18.229982 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:18.230386 22951 ts_manager.cc:194] Registered new tserver with Master: d339073cba9e4cf29879b84ad1715bfe (127.20.65.66:37413)
I20240502 01:53:18.230736 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/ts-2-root/instance:
uuid: "764922bf524647cea3d4612303044281"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-7cj2"
I20240502 01:53:18.231433 22951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33148
I20240502 01:53:18.233325 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:18.234678 23164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:18.234966 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:53:18.235045 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/ts-2-root
uuid: "764922bf524647cea3d4612303044281"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-7cj2"
I20240502 01:53:18.235124 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614741486931-20741-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:18.242048 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:18.242504 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:18.242969 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:18.243672 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:18.243721 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:18.243762 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:18.243789 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:18.255545 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.67:46053
I20240502 01:53:18.255573 23226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.67:46053 every 8 connection(s)
I20240502 01:53:18.261718 23227 heartbeater.cc:347] Connected to a master server at 127.20.65.126:35019
I20240502 01:53:18.261847 23227 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:18.262195 23227 heartbeater.cc:511] Master 127.20.65.126:35019 requested a full tablet report, sending...
I20240502 01:53:18.263036 22951 ts_manager.cc:194] Registered new tserver with Master: 764922bf524647cea3d4612303044281 (127.20.65.67:46053)
I20240502 01:53:18.263247 20741 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006836761s
I20240502 01:53:18.264008 22951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33154
I20240502 01:53:19.181934 23078 heartbeater.cc:503] Master 127.20.65.126:35019 was elected leader, sending a full tablet report...
I20240502 01:53:19.232663 23153 heartbeater.cc:503] Master 127.20.65.126:35019 was elected leader, sending a full tablet report...
I20240502 01:53:19.265262 23227 heartbeater.cc:503] Master 127.20.65.126:35019 was elected leader, sending a full tablet report...
I20240502 01:53:20.058509 23004 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:22.058936 23004 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:24.059353 23004 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:26.059912 23004 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:28.060359 23004 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:30.060851 23004 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:30.344764 20741 tablet_server.cc:178] TabletServer@127.20.65.65:0 shutting down...
I20240502 01:53:30.357895 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:53:30.371956 20741 tablet_server.cc:194] TabletServer@127.20.65.65:0 shutdown complete.
I20240502 01:53:30.374869 20741 tablet_server.cc:178] TabletServer@127.20.65.66:0 shutting down...
I20240502 01:53:30.383018 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:53:30.395581 20741 tablet_server.cc:194] TabletServer@127.20.65.66:0 shutdown complete.
I20240502 01:53:30.398217 20741 tablet_server.cc:178] TabletServer@127.20.65.67:0 shutting down...
I20240502 01:53:30.406514 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:53:30.419036 20741 tablet_server.cc:194] TabletServer@127.20.65.67:0 shutdown complete.
I20240502 01:53:30.421788 20741 master.cc:537] Master@127.20.65.126:35019 shutting down...
I20240502 01:53:30.429744 20741 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:30.429980 20741 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:30.430106 20741 tablet_replica.cc:330] T 00000000000000000000000000000000 P 20f234f048d347e48c9501301d0dbc75: stopping tablet replica
I20240502 01:53:30.443023 20741 master.cc:552] Master@127.20.65.126:35019 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12458 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240502 01:53:30.452289 20741 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.65.126:45287
I20240502 01:53:30.452762 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:30.455413 23231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:30.455695 23232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:30.456552 23234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:30.457054 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:30.457285 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:30.457342 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:30.457386 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614810457370 us; error 0 us; skew 500 ppm
I20240502 01:53:30.457530 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:30.465688 20741 webserver.cc:460] Webserver started at http://127.20.65.126:36939/ using document root <none> and password file <none>
I20240502 01:53:30.465915 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:30.465983 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:30.466092 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:30.466764 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/master-0-root/instance:
uuid: "8228bb81c6b04c29aa83e18e1a29aefe"
format_stamp: "Formatted at 2024-05-02 01:53:30 on dist-test-slave-7cj2"
I20240502 01:53:30.469259 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:30.470795 23239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:30.471129 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:53:30.471238 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/master-0-root
uuid: "8228bb81c6b04c29aa83e18e1a29aefe"
format_stamp: "Formatted at 2024-05-02 01:53:30 on dist-test-slave-7cj2"
I20240502 01:53:30.471351 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:30.485085 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:30.485631 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:30.496940 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.126:45287
I20240502 01:53:30.496959 23291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.126:45287 every 8 connection(s)
I20240502 01:53:30.498203 23292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.502226 23292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe: Bootstrap starting.
I20240502 01:53:30.503436 23292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:30.504900 23292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe: No bootstrap required, opened a new log
I20240502 01:53:30.505457 23292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8228bb81c6b04c29aa83e18e1a29aefe" member_type: VOTER }
I20240502 01:53:30.505590 23292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:30.505668 23292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8228bb81c6b04c29aa83e18e1a29aefe, State: Initialized, Role: FOLLOWER
I20240502 01:53:30.505877 23292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8228bb81c6b04c29aa83e18e1a29aefe" member_type: VOTER }
I20240502 01:53:30.506003 23292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:53:30.506081 23292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:53:30.506172 23292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:30.507900 23292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8228bb81c6b04c29aa83e18e1a29aefe" member_type: VOTER }
I20240502 01:53:30.508087 23292 leader_election.cc:302] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8228bb81c6b04c29aa83e18e1a29aefe; no voters:
I20240502 01:53:30.508459 23292 leader_election.cc:288] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:53:30.508652 23295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:30.509127 23295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [term 1 LEADER]: Becoming Leader. State: Replica: 8228bb81c6b04c29aa83e18e1a29aefe, State: Running, Role: LEADER
I20240502 01:53:30.509303 23292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:53:30.509474 23295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8228bb81c6b04c29aa83e18e1a29aefe" member_type: VOTER }
I20240502 01:53:30.510556 23297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8228bb81c6b04c29aa83e18e1a29aefe. Latest consensus state: current_term: 1 leader_uuid: "8228bb81c6b04c29aa83e18e1a29aefe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8228bb81c6b04c29aa83e18e1a29aefe" member_type: VOTER } }
I20240502 01:53:30.510785 23297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:30.510507 23296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8228bb81c6b04c29aa83e18e1a29aefe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8228bb81c6b04c29aa83e18e1a29aefe" member_type: VOTER } }
I20240502 01:53:30.511094 23296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:30.512156 23300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:53:30.513628 23300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:53:30.514920 20741 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:53:30.516924 23300 catalog_manager.cc:1347] Generated new cluster ID: 66725306de9a400999d598c4ace27ea7
I20240502 01:53:30.517005 23300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:53:30.522111 23300 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:53:30.522882 23300 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:53:30.530154 23300 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe: Generated new TSK 0
I20240502 01:53:30.530392 23300 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:53:30.547680 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:30.550549 23313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:30.550813 23314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:30.551752 23316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:30.552296 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:30.552562 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:30.552645 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:30.552702 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614810552682 us; error 0 us; skew 500 ppm
I20240502 01:53:30.552846 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:30.554144 20741 webserver.cc:460] Webserver started at http://127.20.65.65:33153/ using document root <none> and password file <none>
I20240502 01:53:30.554366 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:30.554472 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:30.554625 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:30.555428 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-0-root/instance:
uuid: "f40c2474fa3547bdb74cb25cf07ab878"
format_stamp: "Formatted at 2024-05-02 01:53:30 on dist-test-slave-7cj2"
I20240502 01:53:30.557947 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:30.559587 23321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:30.559912 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:53:30.560022 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-0-root
uuid: "f40c2474fa3547bdb74cb25cf07ab878"
format_stamp: "Formatted at 2024-05-02 01:53:30 on dist-test-slave-7cj2"
I20240502 01:53:30.560142 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:30.584137 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:30.584669 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:30.585187 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:30.586026 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:30.586099 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:30.586197 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:30.586253 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:30.608412 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.65:36603
I20240502 01:53:30.608429 23384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.65:36603 every 8 connection(s)
I20240502 01:53:30.609956 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:30.612978 23389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:30.613160 23390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:30.613965 23392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:30.614068 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:30.614533 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:30.614593 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:30.614617 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614810614613 us; error 0 us; skew 500 ppm
I20240502 01:53:30.614745 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:30.616405 20741 webserver.cc:460] Webserver started at http://127.20.65.66:34889/ using document root <none> and password file <none>
I20240502 01:53:30.616601 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:30.616665 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:30.616778 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:30.616910 23385 heartbeater.cc:347] Connected to a master server at 127.20.65.126:45287
I20240502 01:53:30.617033 23385 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:30.617326 23385 heartbeater.cc:511] Master 127.20.65.126:45287 requested a full tablet report, sending...
I20240502 01:53:30.617422 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-1-root/instance:
uuid: "68f308269187438f9d121f6ad4be982e"
format_stamp: "Formatted at 2024-05-02 01:53:30 on dist-test-slave-7cj2"
I20240502 01:53:30.618152 23257 ts_manager.cc:194] Registered new tserver with Master: f40c2474fa3547bdb74cb25cf07ab878 (127.20.65.65:36603)
I20240502 01:53:30.619149 23257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35008
I20240502 01:53:30.619966 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:30.621451 23397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:30.621747 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:53:30.621830 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-1-root
uuid: "68f308269187438f9d121f6ad4be982e"
format_stamp: "Formatted at 2024-05-02 01:53:30 on dist-test-slave-7cj2"
I20240502 01:53:30.621922 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:30.627348 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:30.627723 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:30.628127 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:30.628805 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:30.628855 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:30.628917 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:30.628952 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:30.639559 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.66:37707
I20240502 01:53:30.639601 23459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.66:37707 every 8 connection(s)
I20240502 01:53:30.640930 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:30.643842 23464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:30.644337 23463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:30.644604 20741 server_base.cc:1034] running on GCE node
W20240502 01:53:30.645712 23466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:30.646101 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:30.646171 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:30.646211 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614810646208 us; error 0 us; skew 500 ppm
I20240502 01:53:30.646330 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:30.646914 23460 heartbeater.cc:347] Connected to a master server at 127.20.65.126:45287
I20240502 01:53:30.647063 23460 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:30.647449 23460 heartbeater.cc:511] Master 127.20.65.126:45287 requested a full tablet report, sending...
I20240502 01:53:30.647657 20741 webserver.cc:460] Webserver started at http://127.20.65.67:46271/ using document root <none> and password file <none>
I20240502 01:53:30.647868 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:30.647934 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:30.648047 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:30.648312 23257 ts_manager.cc:194] Registered new tserver with Master: 68f308269187438f9d121f6ad4be982e (127.20.65.66:37707)
I20240502 01:53:30.648628 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-2-root/instance:
uuid: "d3fc813f6d1b4306b355f17b90869a4f"
format_stamp: "Formatted at 2024-05-02 01:53:30 on dist-test-slave-7cj2"
I20240502 01:53:30.648998 23257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35016
I20240502 01:53:30.651110 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 01:53:30.652480 23471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:30.652768 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 01:53:30.652855 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-2-root
uuid: "d3fc813f6d1b4306b355f17b90869a4f"
format_stamp: "Formatted at 2024-05-02 01:53:30 on dist-test-slave-7cj2"
I20240502 01:53:30.652940 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:30.666054 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:30.666556 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:30.667008 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:30.667729 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:30.667783 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:30.667853 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:30.667887 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:30.679235 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.67:34417
I20240502 01:53:30.679268 23534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.67:34417 every 8 connection(s)
I20240502 01:53:30.680600 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:30.683377 23538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:30.683773 23539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:30.685148 23541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:30.685420 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:30.685704 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:30.685760 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:30.685822 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614810685816 us; error 0 us; skew 500 ppm
I20240502 01:53:30.685964 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:30.686442 23535 heartbeater.cc:347] Connected to a master server at 127.20.65.126:45287
I20240502 01:53:30.686570 23535 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:30.686945 23535 heartbeater.cc:511] Master 127.20.65.126:45287 requested a full tablet report, sending...
I20240502 01:53:30.687299 20741 webserver.cc:460] Webserver started at http://127.20.65.68:32781/ using document root <none> and password file <none>
I20240502 01:53:30.687513 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:30.687577 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:30.687685 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:30.687760 23257 ts_manager.cc:194] Registered new tserver with Master: d3fc813f6d1b4306b355f17b90869a4f (127.20.65.67:34417)
I20240502 01:53:30.688277 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-3-root/instance:
uuid: "700c4cc7278f474aa02e2942e04a7304"
format_stamp: "Formatted at 2024-05-02 01:53:30 on dist-test-slave-7cj2"
I20240502 01:53:30.688447 23257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35024
I20240502 01:53:30.690580 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:30.691955 23546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:30.692234 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:53:30.692330 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-3-root
uuid: "700c4cc7278f474aa02e2942e04a7304"
format_stamp: "Formatted at 2024-05-02 01:53:30 on dist-test-slave-7cj2"
I20240502 01:53:30.692418 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614741486931-20741-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:30.697072 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:30.697548 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:30.698025 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:30.698760 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:30.698813 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:30.698887 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:30.698926 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:30.710743 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.68:41049
I20240502 01:53:30.710770 23609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.68:41049 every 8 connection(s)
I20240502 01:53:30.716892 23610 heartbeater.cc:347] Connected to a master server at 127.20.65.126:45287
I20240502 01:53:30.717000 23610 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:30.717330 23610 heartbeater.cc:511] Master 127.20.65.126:45287 requested a full tablet report, sending...
I20240502 01:53:30.718075 23257 ts_manager.cc:194] Registered new tserver with Master: 700c4cc7278f474aa02e2942e04a7304 (127.20.65.68:41049)
I20240502 01:53:30.718303 20741 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.006793032s
I20240502 01:53:30.718827 23257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35038
I20240502 01:53:31.620666 23385 heartbeater.cc:503] Master 127.20.65.126:45287 was elected leader, sending a full tablet report...
I20240502 01:53:31.650317 23460 heartbeater.cc:503] Master 127.20.65.126:45287 was elected leader, sending a full tablet report...
I20240502 01:53:31.689507 23535 heartbeater.cc:503] Master 127.20.65.126:45287 was elected leader, sending a full tablet report...
I20240502 01:53:31.720021 23610 heartbeater.cc:503] Master 127.20.65.126:45287 was elected leader, sending a full tablet report...
I20240502 01:53:31.752552 20741 test_util.cc:268] Using random seed: -557320114
I20240502 01:53:31.763428 23257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35044:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 01:53:31.779743 23350 tablet_service.cc:1467] Processing CreateTablet for tablet 90fa95e4fd994dc38448f238dfbaab12 (DEFAULT_TABLE table=test-workload [id=48e618f95dd542a09b7269f711c59581]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:31.780210 23350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90fa95e4fd994dc38448f238dfbaab12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:31.780781 23500 tablet_service.cc:1467] Processing CreateTablet for tablet 90fa95e4fd994dc38448f238dfbaab12 (DEFAULT_TABLE table=test-workload [id=48e618f95dd542a09b7269f711c59581]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:31.781093 23500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90fa95e4fd994dc38448f238dfbaab12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:31.781912 23574 tablet_service.cc:1467] Processing CreateTablet for tablet 90fa95e4fd994dc38448f238dfbaab12 (DEFAULT_TABLE table=test-workload [id=48e618f95dd542a09b7269f711c59581]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:31.782222 23574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90fa95e4fd994dc38448f238dfbaab12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:31.786854 23631 tablet_bootstrap.cc:492] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878: Bootstrap starting.
I20240502 01:53:31.787503 23632 tablet_bootstrap.cc:492] T 90fa95e4fd994dc38448f238dfbaab12 P 700c4cc7278f474aa02e2942e04a7304: Bootstrap starting.
I20240502 01:53:31.788633 23631 tablet_bootstrap.cc:654] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:31.789878 23632 tablet_bootstrap.cc:654] T 90fa95e4fd994dc38448f238dfbaab12 P 700c4cc7278f474aa02e2942e04a7304: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:31.791885 23633 tablet_bootstrap.cc:492] T 90fa95e4fd994dc38448f238dfbaab12 P d3fc813f6d1b4306b355f17b90869a4f: Bootstrap starting.
I20240502 01:53:31.792277 23632 tablet_bootstrap.cc:492] T 90fa95e4fd994dc38448f238dfbaab12 P 700c4cc7278f474aa02e2942e04a7304: No bootstrap required, opened a new log
I20240502 01:53:31.792410 23632 ts_tablet_manager.cc:1367] T 90fa95e4fd994dc38448f238dfbaab12 P 700c4cc7278f474aa02e2942e04a7304: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240502 01:53:31.793161 23633 tablet_bootstrap.cc:654] T 90fa95e4fd994dc38448f238dfbaab12 P d3fc813f6d1b4306b355f17b90869a4f: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:31.793077 23632 raft_consensus.cc:348] T 90fa95e4fd994dc38448f238dfbaab12 P 700c4cc7278f474aa02e2942e04a7304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c2474fa3547bdb74cb25cf07ab878" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 36603 } } peers { permanent_uuid: "700c4cc7278f474aa02e2942e04a7304" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 41049 } } peers { permanent_uuid: "d3fc813f6d1b4306b355f17b90869a4f" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 34417 } }
I20240502 01:53:31.793275 23632 raft_consensus.cc:374] T 90fa95e4fd994dc38448f238dfbaab12 P 700c4cc7278f474aa02e2942e04a7304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:31.793349 23632 raft_consensus.cc:729] T 90fa95e4fd994dc38448f238dfbaab12 P 700c4cc7278f474aa02e2942e04a7304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 700c4cc7278f474aa02e2942e04a7304, State: Initialized, Role: FOLLOWER
I20240502 01:53:31.793577 23632 consensus_queue.cc:260] T 90fa95e4fd994dc38448f238dfbaab12 P 700c4cc7278f474aa02e2942e04a7304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c2474fa3547bdb74cb25cf07ab878" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 36603 } } peers { permanent_uuid: "700c4cc7278f474aa02e2942e04a7304" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 41049 } } peers { permanent_uuid: "d3fc813f6d1b4306b355f17b90869a4f" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 34417 } }
I20240502 01:53:31.793875 23631 tablet_bootstrap.cc:492] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878: No bootstrap required, opened a new log
I20240502 01:53:31.794013 23631 ts_tablet_manager.cc:1367] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240502 01:53:31.794519 23632 ts_tablet_manager.cc:1398] T 90fa95e4fd994dc38448f238dfbaab12 P 700c4cc7278f474aa02e2942e04a7304: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:31.794759 23631 raft_consensus.cc:348] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c2474fa3547bdb74cb25cf07ab878" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 36603 } } peers { permanent_uuid: "700c4cc7278f474aa02e2942e04a7304" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 41049 } } peers { permanent_uuid: "d3fc813f6d1b4306b355f17b90869a4f" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 34417 } }
I20240502 01:53:31.794922 23631 raft_consensus.cc:374] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:31.795022 23631 raft_consensus.cc:729] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40c2474fa3547bdb74cb25cf07ab878, State: Initialized, Role: FOLLOWER
I20240502 01:53:31.795192 23631 consensus_queue.cc:260] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c2474fa3547bdb74cb25cf07ab878" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 36603 } } peers { permanent_uuid: "700c4cc7278f474aa02e2942e04a7304" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 41049 } } peers { permanent_uuid: "d3fc813f6d1b4306b355f17b90869a4f" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 34417 } }
I20240502 01:53:31.795740 23631 ts_tablet_manager.cc:1398] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:31.797361 23633 tablet_bootstrap.cc:492] T 90fa95e4fd994dc38448f238dfbaab12 P d3fc813f6d1b4306b355f17b90869a4f: No bootstrap required, opened a new log
I20240502 01:53:31.797492 23633 ts_tablet_manager.cc:1367] T 90fa95e4fd994dc38448f238dfbaab12 P d3fc813f6d1b4306b355f17b90869a4f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:53:31.798115 23633 raft_consensus.cc:348] T 90fa95e4fd994dc38448f238dfbaab12 P d3fc813f6d1b4306b355f17b90869a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c2474fa3547bdb74cb25cf07ab878" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 36603 } } peers { permanent_uuid: "700c4cc7278f474aa02e2942e04a7304" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 41049 } } peers { permanent_uuid: "d3fc813f6d1b4306b355f17b90869a4f" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 34417 } }
I20240502 01:53:31.798242 23633 raft_consensus.cc:374] T 90fa95e4fd994dc38448f238dfbaab12 P d3fc813f6d1b4306b355f17b90869a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:31.798305 23633 raft_consensus.cc:729] T 90fa95e4fd994dc38448f238dfbaab12 P d3fc813f6d1b4306b355f17b90869a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3fc813f6d1b4306b355f17b90869a4f, State: Initialized, Role: FOLLOWER
I20240502 01:53:31.798528 23633 consensus_queue.cc:260] T 90fa95e4fd994dc38448f238dfbaab12 P d3fc813f6d1b4306b355f17b90869a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c2474fa3547bdb74cb25cf07ab878" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 36603 } } peers { permanent_uuid: "700c4cc7278f474aa02e2942e04a7304" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 41049 } } peers { permanent_uuid: "d3fc813f6d1b4306b355f17b90869a4f" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 34417 } }
I20240502 01:53:31.799060 23633 ts_tablet_manager.cc:1398] T 90fa95e4fd994dc38448f238dfbaab12 P d3fc813f6d1b4306b355f17b90869a4f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:53:31.856688 23637 raft_consensus.cc:482] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:31.856881 23637 raft_consensus.cc:504] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c2474fa3547bdb74cb25cf07ab878" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 36603 } } peers { permanent_uuid: "700c4cc7278f474aa02e2942e04a7304" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 41049 } } peers { permanent_uuid: "d3fc813f6d1b4306b355f17b90869a4f" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 34417 } }
I20240502 01:53:31.857877 23637 leader_election.cc:288] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 700c4cc7278f474aa02e2942e04a7304 (127.20.65.68:41049), d3fc813f6d1b4306b355f17b90869a4f (127.20.65.67:34417)
W20240502 01:53:31.860441 23386 tablet.cc:2369] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:31.864154 23585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90fa95e4fd994dc38448f238dfbaab12" candidate_uuid: "f40c2474fa3547bdb74cb25cf07ab878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "700c4cc7278f474aa02e2942e04a7304" is_pre_election: true
I20240502 01:53:31.864441 23585 raft_consensus.cc:2454] T 90fa95e4fd994dc38448f238dfbaab12 P 700c4cc7278f474aa02e2942e04a7304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f40c2474fa3547bdb74cb25cf07ab878 in term 0.
I20240502 01:53:31.864544 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90fa95e4fd994dc38448f238dfbaab12" candidate_uuid: "f40c2474fa3547bdb74cb25cf07ab878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3fc813f6d1b4306b355f17b90869a4f" is_pre_election: true
I20240502 01:53:31.864781 23510 raft_consensus.cc:2454] T 90fa95e4fd994dc38448f238dfbaab12 P d3fc813f6d1b4306b355f17b90869a4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f40c2474fa3547bdb74cb25cf07ab878 in term 0.
I20240502 01:53:31.864843 23324 leader_election.cc:302] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 700c4cc7278f474aa02e2942e04a7304, f40c2474fa3547bdb74cb25cf07ab878; no voters:
I20240502 01:53:31.865085 23637 raft_consensus.cc:2781] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:31.865164 23637 raft_consensus.cc:482] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:31.865221 23637 raft_consensus.cc:3037] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:31.867116 23637 raft_consensus.cc:504] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c2474fa3547bdb74cb25cf07ab878" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 36603 } } peers { permanent_uuid: "700c4cc7278f474aa02e2942e04a7304" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 41049 } } peers { permanent_uuid: "d3fc813f6d1b4306b355f17b90869a4f" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 34417 } }
I20240502 01:53:31.867591 23637 leader_election.cc:288] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [CANDIDATE]: Term 1 election: Requested vote from peers 700c4cc7278f474aa02e2942e04a7304 (127.20.65.68:41049), d3fc813f6d1b4306b355f17b90869a4f (127.20.65.67:34417)
I20240502 01:53:31.867976 23585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90fa95e4fd994dc38448f238dfbaab12" candidate_uuid: "f40c2474fa3547bdb74cb25cf07ab878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "700c4cc7278f474aa02e2942e04a7304"
I20240502 01:53:31.868026 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90fa95e4fd994dc38448f238dfbaab12" candidate_uuid: "f40c2474fa3547bdb74cb25cf07ab878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3fc813f6d1b4306b355f17b90869a4f"
I20240502 01:53:31.868116 23585 raft_consensus.cc:3037] T 90fa95e4fd994dc38448f238dfbaab12 P 700c4cc7278f474aa02e2942e04a7304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:31.868221 23510 raft_consensus.cc:3037] T 90fa95e4fd994dc38448f238dfbaab12 P d3fc813f6d1b4306b355f17b90869a4f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:31.870049 23585 raft_consensus.cc:2454] T 90fa95e4fd994dc38448f238dfbaab12 P 700c4cc7278f474aa02e2942e04a7304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40c2474fa3547bdb74cb25cf07ab878 in term 1.
I20240502 01:53:31.870074 23510 raft_consensus.cc:2454] T 90fa95e4fd994dc38448f238dfbaab12 P d3fc813f6d1b4306b355f17b90869a4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40c2474fa3547bdb74cb25cf07ab878 in term 1.
I20240502 01:53:31.870360 23324 leader_election.cc:302] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3fc813f6d1b4306b355f17b90869a4f, f40c2474fa3547bdb74cb25cf07ab878; no voters:
I20240502 01:53:31.870611 23637 raft_consensus.cc:2781] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:31.870982 23637 raft_consensus.cc:686] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [term 1 LEADER]: Becoming Leader. State: Replica: f40c2474fa3547bdb74cb25cf07ab878, State: Running, Role: LEADER
I20240502 01:53:31.871208 23637 consensus_queue.cc:237] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c2474fa3547bdb74cb25cf07ab878" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 36603 } } peers { permanent_uuid: "700c4cc7278f474aa02e2942e04a7304" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 41049 } } peers { permanent_uuid: "d3fc813f6d1b4306b355f17b90869a4f" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 34417 } }
I20240502 01:53:31.873697 23257 catalog_manager.cc:5484] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 reported cstate change: term changed from 0 to 1, leader changed from <none> to f40c2474fa3547bdb74cb25cf07ab878 (127.20.65.65). New cstate: current_term: 1 leader_uuid: "f40c2474fa3547bdb74cb25cf07ab878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c2474fa3547bdb74cb25cf07ab878" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 36603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "700c4cc7278f474aa02e2942e04a7304" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 41049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fc813f6d1b4306b355f17b90869a4f" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 34417 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:32.389225 23644 consensus_queue.cc:1035] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fc813f6d1b4306b355f17b90869a4f" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 34417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:32.394004 23644 consensus_queue.cc:1035] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "700c4cc7278f474aa02e2942e04a7304" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 41049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:32.514914 23310 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:32.515367 23310 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:53:32.515470 23310 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:34.515879 23310 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:34.516405 23310 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:53:34.516512 23310 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:36.516896 23310 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:36.517346 23310 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:53:36.517457 23310 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:38.517899 23310 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:38.518419 23310 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:53:38.518543 23310 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:40.518935 23310 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:40.519428 23310 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:53:40.519521 23310 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:42.519905 23310 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:42.520396 23310 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:53:42.520499 23310 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:42.953950 20741 tablet_server.cc:178] TabletServer@127.20.65.65:0 shutting down...
I20240502 01:53:42.966504 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:53:42.966799 20741 tablet_replica.cc:330] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878: stopping tablet replica
I20240502 01:53:42.966979 20741 raft_consensus.cc:2229] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:42.967320 20741 raft_consensus.cc:2258] T 90fa95e4fd994dc38448f238dfbaab12 P f40c2474fa3547bdb74cb25cf07ab878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:42.981109 20741 tablet_server.cc:194] TabletServer@127.20.65.65:0 shutdown complete.
I20240502 01:53:42.983989 20741 tablet_server.cc:178] TabletServer@127.20.65.66:0 shutting down...
I20240502 01:53:42.992080 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:53:43.004690 20741 tablet_server.cc:194] TabletServer@127.20.65.66:0 shutdown complete.
I20240502 01:53:43.007185 20741 tablet_server.cc:178] TabletServer@127.20.65.67:0 shutting down...
I20240502 01:53:43.014771 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:53:43.014971 20741 tablet_replica.cc:330] T 90fa95e4fd994dc38448f238dfbaab12 P d3fc813f6d1b4306b355f17b90869a4f: stopping tablet replica
I20240502 01:53:43.015161 20741 raft_consensus.cc:2229] T 90fa95e4fd994dc38448f238dfbaab12 P d3fc813f6d1b4306b355f17b90869a4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:43.015318 20741 raft_consensus.cc:2258] T 90fa95e4fd994dc38448f238dfbaab12 P d3fc813f6d1b4306b355f17b90869a4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:43.028335 20741 tablet_server.cc:194] TabletServer@127.20.65.67:0 shutdown complete.
I20240502 01:53:43.031003 20741 tablet_server.cc:178] TabletServer@127.20.65.68:0 shutting down...
I20240502 01:53:43.049226 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:53:43.049432 20741 tablet_replica.cc:330] T 90fa95e4fd994dc38448f238dfbaab12 P 700c4cc7278f474aa02e2942e04a7304: stopping tablet replica
I20240502 01:53:43.049592 20741 raft_consensus.cc:2229] T 90fa95e4fd994dc38448f238dfbaab12 P 700c4cc7278f474aa02e2942e04a7304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:43.049764 20741 raft_consensus.cc:2258] T 90fa95e4fd994dc38448f238dfbaab12 P 700c4cc7278f474aa02e2942e04a7304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:43.062991 20741 tablet_server.cc:194] TabletServer@127.20.65.68:0 shutdown complete.
I20240502 01:53:43.065841 20741 master.cc:537] Master@127.20.65.126:45287 shutting down...
I20240502 01:53:43.073238 20741 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:43.073433 20741 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:43.073526 20741 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8228bb81c6b04c29aa83e18e1a29aefe: stopping tablet replica
I20240502 01:53:43.086422 20741 master.cc:552] Master@127.20.65.126:45287 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12645 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240502 01:53:43.097658 20741 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.65.126:33745
I20240502 01:53:43.098115 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:43.100522 23655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:43.100869 23656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:43.101536 23658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:43.101660 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:43.101984 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:43.102038 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:43.102072 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614823102067 us; error 0 us; skew 500 ppm
I20240502 01:53:43.102187 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:43.103396 20741 webserver.cc:460] Webserver started at http://127.20.65.126:36901/ using document root <none> and password file <none>
I20240502 01:53:43.103574 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:43.103639 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:43.103730 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:43.104300 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/master-0-root/instance:
uuid: "80a65391cf194c349b952d500210039b"
format_stamp: "Formatted at 2024-05-02 01:53:43 on dist-test-slave-7cj2"
I20240502 01:53:43.106763 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:53:43.108235 23663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:43.108546 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:53:43.108651 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/master-0-root
uuid: "80a65391cf194c349b952d500210039b"
format_stamp: "Formatted at 2024-05-02 01:53:43 on dist-test-slave-7cj2"
I20240502 01:53:43.108753 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:43.128430 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:43.129160 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:43.142562 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.126:33745
I20240502 01:53:43.142577 23714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.126:33745 every 8 connection(s)
I20240502 01:53:43.143877 23715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:43.148129 23715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b: Bootstrap starting.
I20240502 01:53:43.149281 23715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:43.150784 23715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b: No bootstrap required, opened a new log
I20240502 01:53:43.151338 23715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a65391cf194c349b952d500210039b" member_type: VOTER }
I20240502 01:53:43.151470 23715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:43.151537 23715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a65391cf194c349b952d500210039b, State: Initialized, Role: FOLLOWER
I20240502 01:53:43.151744 23715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a65391cf194c349b952d500210039b" member_type: VOTER }
I20240502 01:53:43.151870 23715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:53:43.151932 23715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:53:43.152001 23715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:43.153689 23715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a65391cf194c349b952d500210039b" member_type: VOTER }
I20240502 01:53:43.153870 23715 leader_election.cc:302] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80a65391cf194c349b952d500210039b; no voters:
I20240502 01:53:43.154210 23715 leader_election.cc:288] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:53:43.154325 23718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:43.154806 23718 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [term 1 LEADER]: Becoming Leader. State: Replica: 80a65391cf194c349b952d500210039b, State: Running, Role: LEADER
I20240502 01:53:43.155071 23715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:53:43.155033 23718 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a65391cf194c349b952d500210039b" member_type: VOTER }
I20240502 01:53:43.155900 23720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80a65391cf194c349b952d500210039b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a65391cf194c349b952d500210039b" member_type: VOTER } }
I20240502 01:53:43.155959 23721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80a65391cf194c349b952d500210039b. Latest consensus state: current_term: 1 leader_uuid: "80a65391cf194c349b952d500210039b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a65391cf194c349b952d500210039b" member_type: VOTER } }
I20240502 01:53:43.156085 23720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:43.156109 23721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:43.157096 23724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:53:43.158722 23724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:53:43.160086 20741 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:53:43.161505 23724 catalog_manager.cc:1347] Generated new cluster ID: f52043dd8f1f40ca8d7916a7648def5e
I20240502 01:53:43.161578 23724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:53:43.168254 23724 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:53:43.168969 23724 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:53:43.178191 23724 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b: Generated new TSK 0
I20240502 01:53:43.178426 23724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:53:43.192824 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:43.195447 23737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:43.195734 23738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:43.196233 23740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:43.196408 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:43.196681 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:43.196735 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:43.196769 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614823196766 us; error 0 us; skew 500 ppm
I20240502 01:53:43.196868 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:43.198020 20741 webserver.cc:460] Webserver started at http://127.20.65.65:42473/ using document root <none> and password file <none>
I20240502 01:53:43.198204 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:43.198261 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:43.198352 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:43.198990 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/ts-0-root/instance:
uuid: "3cc71c76c3dd4bcb89175737e958b6ac"
format_stamp: "Formatted at 2024-05-02 01:53:43 on dist-test-slave-7cj2"
I20240502 01:53:43.201380 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:53:43.202944 23745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:43.203236 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:53:43.203341 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/ts-0-root
uuid: "3cc71c76c3dd4bcb89175737e958b6ac"
format_stamp: "Formatted at 2024-05-02 01:53:43 on dist-test-slave-7cj2"
I20240502 01:53:43.203441 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:43.220939 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:43.221402 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:43.221884 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:43.222750 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:43.222807 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:43.222888 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:43.222925 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:43.234202 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.65:45609
I20240502 01:53:43.234225 23807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.65:45609 every 8 connection(s)
I20240502 01:53:43.235687 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:43.238417 23812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:43.238765 23813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:43.239151 23815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:43.239635 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:43.239903 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:43.239951 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:43.239972 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614823239969 us; error 0 us; skew 500 ppm
I20240502 01:53:43.240088 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:43.241950 20741 webserver.cc:460] Webserver started at http://127.20.65.66:41089/ using document root <none> and password file <none>
I20240502 01:53:43.242177 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:43.242237 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:43.242357 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:43.242429 23808 heartbeater.cc:347] Connected to a master server at 127.20.65.126:33745
I20240502 01:53:43.242556 23808 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:43.242909 23808 heartbeater.cc:511] Master 127.20.65.126:33745 requested a full tablet report, sending...
I20240502 01:53:43.243103 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/ts-1-root/instance:
uuid: "d0a2baf23840409dafe0664025a2f57a"
format_stamp: "Formatted at 2024-05-02 01:53:43 on dist-test-slave-7cj2"
I20240502 01:53:43.243762 23680 ts_manager.cc:194] Registered new tserver with Master: 3cc71c76c3dd4bcb89175737e958b6ac (127.20.65.65:45609)
I20240502 01:53:43.244771 23680 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51038
I20240502 01:53:43.245714 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:43.247227 23820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:43.247514 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:53:43.247596 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/ts-1-root
uuid: "d0a2baf23840409dafe0664025a2f57a"
format_stamp: "Formatted at 2024-05-02 01:53:43 on dist-test-slave-7cj2"
I20240502 01:53:43.247691 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:43.260516 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:43.260934 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:43.261353 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:43.262097 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:43.262152 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:43.262214 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:43.262249 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:43.273481 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.66:43623
I20240502 01:53:43.273505 23883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.66:43623 every 8 connection(s)
I20240502 01:53:43.274893 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:43.277606 23887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:43.277877 23888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:43.278436 20741 server_base.cc:1034] running on GCE node
W20240502 01:53:43.278754 23890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:43.279167 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:43.279232 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:43.279282 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614823279278 us; error 0 us; skew 500 ppm
I20240502 01:53:43.279417 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:43.280927 20741 webserver.cc:460] Webserver started at http://127.20.65.67:43667/ using document root <none> and password file <none>
I20240502 01:53:43.281159 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:43.281220 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:43.281324 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:43.281338 23884 heartbeater.cc:347] Connected to a master server at 127.20.65.126:33745
I20240502 01:53:43.281445 23884 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:43.281701 23884 heartbeater.cc:511] Master 127.20.65.126:33745 requested a full tablet report, sending...
I20240502 01:53:43.281917 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/ts-2-root/instance:
uuid: "a9c4c073d7034491bd489135929e1f3b"
format_stamp: "Formatted at 2024-05-02 01:53:43 on dist-test-slave-7cj2"
I20240502 01:53:43.282618 23680 ts_manager.cc:194] Registered new tserver with Master: d0a2baf23840409dafe0664025a2f57a (127.20.65.66:43623)
I20240502 01:53:43.283327 23680 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51052
I20240502 01:53:43.284543 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20240502 01:53:43.286162 23895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:43.286548 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 01:53:43.286644 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/ts-2-root
uuid: "a9c4c073d7034491bd489135929e1f3b"
format_stamp: "Formatted at 2024-05-02 01:53:43 on dist-test-slave-7cj2"
I20240502 01:53:43.286744 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614741486931-20741-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:43.298465 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:43.298887 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:43.299342 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:43.300040 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:43.300093 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:43.300158 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:43.300194 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:43.311755 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.67:40273
I20240502 01:53:43.311806 23958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.67:40273 every 8 connection(s)
I20240502 01:53:43.317979 23959 heartbeater.cc:347] Connected to a master server at 127.20.65.126:33745
I20240502 01:53:43.318114 23959 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:43.318485 23959 heartbeater.cc:511] Master 127.20.65.126:33745 requested a full tablet report, sending...
I20240502 01:53:43.319278 23680 ts_manager.cc:194] Registered new tserver with Master: a9c4c073d7034491bd489135929e1f3b (127.20.65.67:40273)
I20240502 01:53:43.319451 20741 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00684719s
I20240502 01:53:43.320026 23680 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51066
I20240502 01:53:44.246217 23808 heartbeater.cc:503] Master 127.20.65.126:33745 was elected leader, sending a full tablet report...
I20240502 01:53:44.284440 23884 heartbeater.cc:503] Master 127.20.65.126:33745 was elected leader, sending a full tablet report...
I20240502 01:53:44.321265 23959 heartbeater.cc:503] Master 127.20.65.126:33745 was elected leader, sending a full tablet report...
I20240502 01:53:44.353801 20741 test_util.cc:268] Using random seed: -544718883
I20240502 01:53:44.364807 23680 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51084:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:53:44.365345 23680 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:53:44.383685 23773 tablet_service.cc:1467] Processing CreateTablet for tablet b3b56d69b1b943a2bb904be4628be047 (DEFAULT_TABLE table=test-workload [id=f9beeb1523ea479e8841f28637c563c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.384210 23773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b56d69b1b943a2bb904be4628be047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.391206 23980 tablet_bootstrap.cc:492] T b3b56d69b1b943a2bb904be4628be047 P 3cc71c76c3dd4bcb89175737e958b6ac: Bootstrap starting.
I20240502 01:53:44.393095 23980 tablet_bootstrap.cc:654] T b3b56d69b1b943a2bb904be4628be047 P 3cc71c76c3dd4bcb89175737e958b6ac: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.394734 23848 tablet_service.cc:1467] Processing CreateTablet for tablet b3b56d69b1b943a2bb904be4628be047 (DEFAULT_TABLE table=test-workload [id=f9beeb1523ea479e8841f28637c563c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.395126 23848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b56d69b1b943a2bb904be4628be047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.395408 23923 tablet_service.cc:1467] Processing CreateTablet for tablet b3b56d69b1b943a2bb904be4628be047 (DEFAULT_TABLE table=test-workload [id=f9beeb1523ea479e8841f28637c563c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.395974 23923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b56d69b1b943a2bb904be4628be047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.400521 23980 tablet_bootstrap.cc:492] T b3b56d69b1b943a2bb904be4628be047 P 3cc71c76c3dd4bcb89175737e958b6ac: No bootstrap required, opened a new log
I20240502 01:53:44.400696 23980 ts_tablet_manager.cc:1367] T b3b56d69b1b943a2bb904be4628be047 P 3cc71c76c3dd4bcb89175737e958b6ac: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 01:53:44.401083 23982 tablet_bootstrap.cc:492] T b3b56d69b1b943a2bb904be4628be047 P d0a2baf23840409dafe0664025a2f57a: Bootstrap starting.
I20240502 01:53:44.401496 23980 raft_consensus.cc:348] T b3b56d69b1b943a2bb904be4628be047 P 3cc71c76c3dd4bcb89175737e958b6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc71c76c3dd4bcb89175737e958b6ac" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 45609 } } peers { permanent_uuid: "a9c4c073d7034491bd489135929e1f3b" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 40273 } } peers { permanent_uuid: "d0a2baf23840409dafe0664025a2f57a" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 43623 } }
I20240502 01:53:44.401732 23980 raft_consensus.cc:374] T b3b56d69b1b943a2bb904be4628be047 P 3cc71c76c3dd4bcb89175737e958b6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.401837 23980 raft_consensus.cc:729] T b3b56d69b1b943a2bb904be4628be047 P 3cc71c76c3dd4bcb89175737e958b6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc71c76c3dd4bcb89175737e958b6ac, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.402078 23980 consensus_queue.cc:260] T b3b56d69b1b943a2bb904be4628be047 P 3cc71c76c3dd4bcb89175737e958b6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc71c76c3dd4bcb89175737e958b6ac" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 45609 } } peers { permanent_uuid: "a9c4c073d7034491bd489135929e1f3b" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 40273 } } peers { permanent_uuid: "d0a2baf23840409dafe0664025a2f57a" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 43623 } }
I20240502 01:53:44.402465 23982 tablet_bootstrap.cc:654] T b3b56d69b1b943a2bb904be4628be047 P d0a2baf23840409dafe0664025a2f57a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.409076 23980 ts_tablet_manager.cc:1398] T b3b56d69b1b943a2bb904be4628be047 P 3cc71c76c3dd4bcb89175737e958b6ac: Time spent starting tablet: real 0.008s user 0.000s sys 0.008s
I20240502 01:53:44.409405 23984 tablet_bootstrap.cc:492] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b: Bootstrap starting.
I20240502 01:53:44.410429 23982 tablet_bootstrap.cc:492] T b3b56d69b1b943a2bb904be4628be047 P d0a2baf23840409dafe0664025a2f57a: No bootstrap required, opened a new log
I20240502 01:53:44.410626 23982 ts_tablet_manager.cc:1367] T b3b56d69b1b943a2bb904be4628be047 P d0a2baf23840409dafe0664025a2f57a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240502 01:53:44.411232 23984 tablet_bootstrap.cc:654] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.411484 23982 raft_consensus.cc:348] T b3b56d69b1b943a2bb904be4628be047 P d0a2baf23840409dafe0664025a2f57a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc71c76c3dd4bcb89175737e958b6ac" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 45609 } } peers { permanent_uuid: "a9c4c073d7034491bd489135929e1f3b" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 40273 } } peers { permanent_uuid: "d0a2baf23840409dafe0664025a2f57a" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 43623 } }
I20240502 01:53:44.411749 23982 raft_consensus.cc:374] T b3b56d69b1b943a2bb904be4628be047 P d0a2baf23840409dafe0664025a2f57a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.411846 23982 raft_consensus.cc:729] T b3b56d69b1b943a2bb904be4628be047 P d0a2baf23840409dafe0664025a2f57a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2baf23840409dafe0664025a2f57a, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.412159 23982 consensus_queue.cc:260] T b3b56d69b1b943a2bb904be4628be047 P d0a2baf23840409dafe0664025a2f57a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc71c76c3dd4bcb89175737e958b6ac" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 45609 } } peers { permanent_uuid: "a9c4c073d7034491bd489135929e1f3b" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 40273 } } peers { permanent_uuid: "d0a2baf23840409dafe0664025a2f57a" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 43623 } }
I20240502 01:53:44.412981 23982 ts_tablet_manager.cc:1398] T b3b56d69b1b943a2bb904be4628be047 P d0a2baf23840409dafe0664025a2f57a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:44.413209 23984 tablet_bootstrap.cc:492] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b: No bootstrap required, opened a new log
I20240502 01:53:44.413375 23984 ts_tablet_manager.cc:1367] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:44.414196 23984 raft_consensus.cc:348] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc71c76c3dd4bcb89175737e958b6ac" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 45609 } } peers { permanent_uuid: "a9c4c073d7034491bd489135929e1f3b" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 40273 } } peers { permanent_uuid: "d0a2baf23840409dafe0664025a2f57a" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 43623 } }
I20240502 01:53:44.414510 23984 raft_consensus.cc:374] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.414657 23984 raft_consensus.cc:729] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c4c073d7034491bd489135929e1f3b, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.414933 23984 consensus_queue.cc:260] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc71c76c3dd4bcb89175737e958b6ac" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 45609 } } peers { permanent_uuid: "a9c4c073d7034491bd489135929e1f3b" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 40273 } } peers { permanent_uuid: "d0a2baf23840409dafe0664025a2f57a" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 43623 } }
I20240502 01:53:44.415675 23984 ts_tablet_manager.cc:1398] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:44.439218 23988 raft_consensus.cc:482] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:44.439361 23988 raft_consensus.cc:504] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc71c76c3dd4bcb89175737e958b6ac" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 45609 } } peers { permanent_uuid: "a9c4c073d7034491bd489135929e1f3b" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 40273 } } peers { permanent_uuid: "d0a2baf23840409dafe0664025a2f57a" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 43623 } }
I20240502 01:53:44.440045 23988 leader_election.cc:288] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cc71c76c3dd4bcb89175737e958b6ac (127.20.65.65:45609), d0a2baf23840409dafe0664025a2f57a (127.20.65.66:43623)
I20240502 01:53:44.446354 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b56d69b1b943a2bb904be4628be047" candidate_uuid: "a9c4c073d7034491bd489135929e1f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc71c76c3dd4bcb89175737e958b6ac" is_pre_election: true
I20240502 01:53:44.446552 23858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b56d69b1b943a2bb904be4628be047" candidate_uuid: "a9c4c073d7034491bd489135929e1f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2baf23840409dafe0664025a2f57a" is_pre_election: true
I20240502 01:53:44.446642 23783 raft_consensus.cc:2454] T b3b56d69b1b943a2bb904be4628be047 P 3cc71c76c3dd4bcb89175737e958b6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9c4c073d7034491bd489135929e1f3b in term 0.
I20240502 01:53:44.446811 23858 raft_consensus.cc:2454] T b3b56d69b1b943a2bb904be4628be047 P d0a2baf23840409dafe0664025a2f57a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9c4c073d7034491bd489135929e1f3b in term 0.
I20240502 01:53:44.447110 23898 leader_election.cc:302] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cc71c76c3dd4bcb89175737e958b6ac, a9c4c073d7034491bd489135929e1f3b; no voters:
I20240502 01:53:44.447374 23988 raft_consensus.cc:2781] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:44.447484 23988 raft_consensus.cc:482] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:44.447561 23988 raft_consensus.cc:3037] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.449433 23988 raft_consensus.cc:504] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc71c76c3dd4bcb89175737e958b6ac" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 45609 } } peers { permanent_uuid: "a9c4c073d7034491bd489135929e1f3b" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 40273 } } peers { permanent_uuid: "d0a2baf23840409dafe0664025a2f57a" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 43623 } }
I20240502 01:53:44.449999 23988 leader_election.cc:288] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [CANDIDATE]: Term 1 election: Requested vote from peers 3cc71c76c3dd4bcb89175737e958b6ac (127.20.65.65:45609), d0a2baf23840409dafe0664025a2f57a (127.20.65.66:43623)
I20240502 01:53:44.450417 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b56d69b1b943a2bb904be4628be047" candidate_uuid: "a9c4c073d7034491bd489135929e1f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc71c76c3dd4bcb89175737e958b6ac"
I20240502 01:53:44.450517 23858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b56d69b1b943a2bb904be4628be047" candidate_uuid: "a9c4c073d7034491bd489135929e1f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2baf23840409dafe0664025a2f57a"
I20240502 01:53:44.450618 23783 raft_consensus.cc:3037] T b3b56d69b1b943a2bb904be4628be047 P 3cc71c76c3dd4bcb89175737e958b6ac [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.450711 23858 raft_consensus.cc:3037] T b3b56d69b1b943a2bb904be4628be047 P d0a2baf23840409dafe0664025a2f57a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.452589 23783 raft_consensus.cc:2454] T b3b56d69b1b943a2bb904be4628be047 P 3cc71c76c3dd4bcb89175737e958b6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9c4c073d7034491bd489135929e1f3b in term 1.
I20240502 01:53:44.452594 23858 raft_consensus.cc:2454] T b3b56d69b1b943a2bb904be4628be047 P d0a2baf23840409dafe0664025a2f57a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9c4c073d7034491bd489135929e1f3b in term 1.
I20240502 01:53:44.452984 23898 leader_election.cc:302] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cc71c76c3dd4bcb89175737e958b6ac, a9c4c073d7034491bd489135929e1f3b; no voters:
I20240502 01:53:44.453403 23988 raft_consensus.cc:2781] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:44.453842 23988 raft_consensus.cc:686] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [term 1 LEADER]: Becoming Leader. State: Replica: a9c4c073d7034491bd489135929e1f3b, State: Running, Role: LEADER
I20240502 01:53:44.454133 23988 consensus_queue.cc:237] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc71c76c3dd4bcb89175737e958b6ac" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 45609 } } peers { permanent_uuid: "a9c4c073d7034491bd489135929e1f3b" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 40273 } } peers { permanent_uuid: "d0a2baf23840409dafe0664025a2f57a" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 43623 } }
I20240502 01:53:44.456733 23679 catalog_manager.cc:5484] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b reported cstate change: term changed from 0 to 1, leader changed from <none> to a9c4c073d7034491bd489135929e1f3b (127.20.65.67). New cstate: current_term: 1 leader_uuid: "a9c4c073d7034491bd489135929e1f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc71c76c3dd4bcb89175737e958b6ac" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 45609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9c4c073d7034491bd489135929e1f3b" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 40273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2baf23840409dafe0664025a2f57a" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 43623 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:44.954278 23988 consensus_queue.cc:1035] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cc71c76c3dd4bcb89175737e958b6ac" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 45609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:44.959371 23988 consensus_queue.cc:1035] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2baf23840409dafe0664025a2f57a" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:45.159577 23729 placement_policy_util.cc:407] tablet b3b56d69b1b943a2bb904be4628be047: detected majority of replicas (2 of 3) at location L0
I20240502 01:53:45.160089 23734 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:45.160529 23734 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:53:45.160629 23734 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:46.162499 23729 placement_policy_util.cc:407] tablet b3b56d69b1b943a2bb904be4628be047: detected majority of replicas (2 of 3) at location L0
I20240502 01:53:47.160987 23734 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:47.161470 23734 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:53:47.161574 23734 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:47.164814 23729 placement_policy_util.cc:407] tablet b3b56d69b1b943a2bb904be4628be047: detected majority of replicas (2 of 3) at location L0
I20240502 01:53:48.167192 23729 placement_policy_util.cc:407] tablet b3b56d69b1b943a2bb904be4628be047: detected majority of replicas (2 of 3) at location L0
I20240502 01:53:49.161957 23734 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:49.162420 23734 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:53:49.162520 23734 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:49.169598 23729 placement_policy_util.cc:407] tablet b3b56d69b1b943a2bb904be4628be047: detected majority of replicas (2 of 3) at location L0
I20240502 01:53:50.171994 23729 placement_policy_util.cc:407] tablet b3b56d69b1b943a2bb904be4628be047: detected majority of replicas (2 of 3) at location L0
I20240502 01:53:51.162954 23734 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:51.163467 23734 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:53:51.163581 23734 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:51.174345 23729 placement_policy_util.cc:407] tablet b3b56d69b1b943a2bb904be4628be047: detected majority of replicas (2 of 3) at location L0
I20240502 01:53:52.176692 23729 placement_policy_util.cc:407] tablet b3b56d69b1b943a2bb904be4628be047: detected majority of replicas (2 of 3) at location L0
I20240502 01:53:53.164050 23734 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:53.164530 23734 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:53:53.164619 23734 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:53.179067 23729 placement_policy_util.cc:407] tablet b3b56d69b1b943a2bb904be4628be047: detected majority of replicas (2 of 3) at location L0
I20240502 01:53:54.181361 23729 placement_policy_util.cc:407] tablet b3b56d69b1b943a2bb904be4628be047: detected majority of replicas (2 of 3) at location L0
I20240502 01:53:55.165026 23734 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:55.165549 23734 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:53:55.165645 23734 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:55.183771 23729 placement_policy_util.cc:407] tablet b3b56d69b1b943a2bb904be4628be047: detected majority of replicas (2 of 3) at location L0
I20240502 01:53:55.530961 20741 tablet_server.cc:178] TabletServer@127.20.65.65:0 shutting down...
I20240502 01:53:55.548744 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:53:55.549095 20741 tablet_replica.cc:330] T b3b56d69b1b943a2bb904be4628be047 P 3cc71c76c3dd4bcb89175737e958b6ac: stopping tablet replica
I20240502 01:53:55.549296 20741 raft_consensus.cc:2229] T b3b56d69b1b943a2bb904be4628be047 P 3cc71c76c3dd4bcb89175737e958b6ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:55.549506 20741 raft_consensus.cc:2258] T b3b56d69b1b943a2bb904be4628be047 P 3cc71c76c3dd4bcb89175737e958b6ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:55.563519 20741 tablet_server.cc:194] TabletServer@127.20.65.65:0 shutdown complete.
I20240502 01:53:55.566358 20741 tablet_server.cc:178] TabletServer@127.20.65.66:0 shutting down...
I20240502 01:53:55.575018 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:53:55.575242 20741 tablet_replica.cc:330] T b3b56d69b1b943a2bb904be4628be047 P d0a2baf23840409dafe0664025a2f57a: stopping tablet replica
I20240502 01:53:55.575423 20741 raft_consensus.cc:2229] T b3b56d69b1b943a2bb904be4628be047 P d0a2baf23840409dafe0664025a2f57a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:55.575608 20741 raft_consensus.cc:2258] T b3b56d69b1b943a2bb904be4628be047 P d0a2baf23840409dafe0664025a2f57a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:55.590799 20741 tablet_server.cc:194] TabletServer@127.20.65.66:0 shutdown complete.
I20240502 01:53:55.593554 20741 tablet_server.cc:178] TabletServer@127.20.65.67:0 shutting down...
I20240502 01:53:55.601626 20741 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:53:55.601883 20741 tablet_replica.cc:330] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b: stopping tablet replica
I20240502 01:53:55.602059 20741 raft_consensus.cc:2229] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:55.602336 20741 raft_consensus.cc:2258] T b3b56d69b1b943a2bb904be4628be047 P a9c4c073d7034491bd489135929e1f3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:55.615466 20741 tablet_server.cc:194] TabletServer@127.20.65.67:0 shutdown complete.
I20240502 01:53:55.618101 20741 master.cc:537] Master@127.20.65.126:33745 shutting down...
I20240502 01:53:55.626366 20741 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:55.626588 20741 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:55.626703 20741 tablet_replica.cc:330] T 00000000000000000000000000000000 P 80a65391cf194c349b952d500210039b: stopping tablet replica
I20240502 01:53:55.639593 20741 master.cc:552] Master@127.20.65.126:33745 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12552 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240502 01:53:55.650606 20741 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.65.126:36935
I20240502 01:53:55.651038 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:55.653618 24004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:55.653846 24003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:55.654325 24006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:55.654975 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:55.655185 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:55.655251 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:55.655310 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614835655292 us; error 0 us; skew 500 ppm
I20240502 01:53:55.655488 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:55.656697 20741 webserver.cc:460] Webserver started at http://127.20.65.126:44979/ using document root <none> and password file <none>
I20240502 01:53:55.656911 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:55.656991 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:55.657130 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:55.657756 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/master-0-root/instance:
uuid: "8370e5269f34454c82b2dccbb340c363"
format_stamp: "Formatted at 2024-05-02 01:53:55 on dist-test-slave-7cj2"
I20240502 01:53:55.660178 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:55.661592 24011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:55.661911 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:53:55.662029 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/master-0-root
uuid: "8370e5269f34454c82b2dccbb340c363"
format_stamp: "Formatted at 2024-05-02 01:53:55 on dist-test-slave-7cj2"
I20240502 01:53:55.662150 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:55.681391 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:55.681895 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:55.692226 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.126:36935
I20240502 01:53:55.692250 24062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.126:36935 every 8 connection(s)
I20240502 01:53:55.693413 24063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.697371 24063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363: Bootstrap starting.
I20240502 01:53:55.698465 24063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.699853 24063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363: No bootstrap required, opened a new log
I20240502 01:53:55.700410 24063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8370e5269f34454c82b2dccbb340c363" member_type: VOTER }
I20240502 01:53:55.700546 24063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:55.700627 24063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8370e5269f34454c82b2dccbb340c363, State: Initialized, Role: FOLLOWER
I20240502 01:53:55.700838 24063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8370e5269f34454c82b2dccbb340c363" member_type: VOTER }
I20240502 01:53:55.700963 24063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:53:55.701043 24063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:53:55.701143 24063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.702862 24063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8370e5269f34454c82b2dccbb340c363" member_type: VOTER }
I20240502 01:53:55.703056 24063 leader_election.cc:302] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8370e5269f34454c82b2dccbb340c363; no voters:
I20240502 01:53:55.703414 24063 leader_election.cc:288] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:53:55.703505 24066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:55.703927 24066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [term 1 LEADER]: Becoming Leader. State: Replica: 8370e5269f34454c82b2dccbb340c363, State: Running, Role: LEADER
I20240502 01:53:55.704149 24066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8370e5269f34454c82b2dccbb340c363" member_type: VOTER }
I20240502 01:53:55.704226 24063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:53:55.704824 24067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8370e5269f34454c82b2dccbb340c363" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8370e5269f34454c82b2dccbb340c363" member_type: VOTER } }
I20240502 01:53:55.704880 24068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8370e5269f34454c82b2dccbb340c363. Latest consensus state: current_term: 1 leader_uuid: "8370e5269f34454c82b2dccbb340c363" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8370e5269f34454c82b2dccbb340c363" member_type: VOTER } }
I20240502 01:53:55.705080 24068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:55.706094 24067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:55.706184 24070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:53:55.707633 24070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:53:55.709944 20741 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:53:55.710597 24070 catalog_manager.cc:1347] Generated new cluster ID: f545b8923c7a4d538fd6a303140ff3d5
I20240502 01:53:55.710686 24070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:53:55.734727 24070 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:53:55.735466 24070 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:53:55.744948 24070 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8370e5269f34454c82b2dccbb340c363: Generated new TSK 0
I20240502 01:53:55.745160 24070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:53:55.774775 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:55.777478 24084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:55.777699 24085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:55.778491 24087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:55.778681 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:55.778882 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:55.778934 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:55.778968 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614835778963 us; error 0 us; skew 500 ppm
I20240502 01:53:55.779100 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:55.780194 20741 webserver.cc:460] Webserver started at http://127.20.65.65:35711/ using document root <none> and password file <none>
I20240502 01:53:55.780405 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:55.780462 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:55.780589 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:55.781189 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-0-root/instance:
uuid: "227af58cb9554c438ae592baff6af4fb"
format_stamp: "Formatted at 2024-05-02 01:53:55 on dist-test-slave-7cj2"
I20240502 01:53:55.783674 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 01:53:55.785099 24092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:55.785419 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 01:53:55.785509 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-0-root
uuid: "227af58cb9554c438ae592baff6af4fb"
format_stamp: "Formatted at 2024-05-02 01:53:55 on dist-test-slave-7cj2"
I20240502 01:53:55.785614 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:55.794996 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:55.795414 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:55.795883 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:55.796622 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:55.796679 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:55.796741 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:55.796777 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:55.807193 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.65:43551
I20240502 01:53:55.807206 24155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.65:43551 every 8 connection(s)
I20240502 01:53:55.808804 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:55.811525 24160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:55.811720 24161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:55.814312 24163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:55.814662 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:55.814924 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 01:53:55.814956 24156 heartbeater.cc:347] Connected to a master server at 127.20.65.126:36935
W20240502 01:53:55.814977 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:55.815060 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614835815055 us; error 0 us; skew 500 ppm
I20240502 01:53:55.815099 24156 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:55.815204 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:55.815423 24156 heartbeater.cc:511] Master 127.20.65.126:36935 requested a full tablet report, sending...
I20240502 01:53:55.816237 24028 ts_manager.cc:194] Registered new tserver with Master: 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551)
I20240502 01:53:55.816442 20741 webserver.cc:460] Webserver started at http://127.20.65.66:39811/ using document root <none> and password file <none>
I20240502 01:53:55.816627 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:55.816689 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:55.816793 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:55.817243 24028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54764
I20240502 01:53:55.817304 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-1-root/instance:
uuid: "9716594f476e436b91b93ceaed4bcf59"
format_stamp: "Formatted at 2024-05-02 01:53:55 on dist-test-slave-7cj2"
I20240502 01:53:55.819576 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:55.820888 24168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:55.821244 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:53:55.821326 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-1-root
uuid: "9716594f476e436b91b93ceaed4bcf59"
format_stamp: "Formatted at 2024-05-02 01:53:55 on dist-test-slave-7cj2"
I20240502 01:53:55.821408 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:55.841560 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:55.841996 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:55.842458 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:55.843143 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:55.843194 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:55.843236 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:55.843276 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:55.853413 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.66:37365
I20240502 01:53:55.853443 24231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.66:37365 every 8 connection(s)
I20240502 01:53:55.854878 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:55.857537 24235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:55.857741 24236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:55.858639 24238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:55.859036 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:55.859340 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:55.859411 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:55.859442 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614835859439 us; error 0 us; skew 500 ppm
I20240502 01:53:55.859591 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:55.860677 24232 heartbeater.cc:347] Connected to a master server at 127.20.65.126:36935
I20240502 01:53:55.860793 24232 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:55.861099 24232 heartbeater.cc:511] Master 127.20.65.126:36935 requested a full tablet report, sending...
I20240502 01:53:55.861474 20741 webserver.cc:460] Webserver started at http://127.20.65.67:35777/ using document root <none> and password file <none>
I20240502 01:53:55.861752 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:55.861832 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:55.861970 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:55.861970 24028 ts_manager.cc:194] Registered new tserver with Master: 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:53:55.862603 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-2-root/instance:
uuid: "a6f86494e4de4d3c851c5dab07b40568"
format_stamp: "Formatted at 2024-05-02 01:53:55 on dist-test-slave-7cj2"
I20240502 01:53:55.862848 24028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54776
I20240502 01:53:55.864944 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:55.866279 24243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:55.866616 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:53:55.866717 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-2-root
uuid: "a6f86494e4de4d3c851c5dab07b40568"
format_stamp: "Formatted at 2024-05-02 01:53:55 on dist-test-slave-7cj2"
I20240502 01:53:55.866811 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:55.885697 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:55.886255 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:55.886786 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:55.887564 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:55.887619 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:55.887702 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:55.887737 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:55.898680 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.67:33253
I20240502 01:53:55.898706 24305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.67:33253 every 8 connection(s)
I20240502 01:53:55.905297 24306 heartbeater.cc:347] Connected to a master server at 127.20.65.126:36935
I20240502 01:53:55.905424 24306 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:55.905745 24306 heartbeater.cc:511] Master 127.20.65.126:36935 requested a full tablet report, sending...
I20240502 01:53:55.906559 24028 ts_manager.cc:194] Registered new tserver with Master: a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:55.907297 24028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54786
I20240502 01:53:55.907297 20741 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007857026s
I20240502 01:53:56.818523 24156 heartbeater.cc:503] Master 127.20.65.126:36935 was elected leader, sending a full tablet report...
I20240502 01:53:56.864015 24232 heartbeater.cc:503] Master 127.20.65.126:36935 was elected leader, sending a full tablet report...
I20240502 01:53:56.908766 24306 heartbeater.cc:503] Master 127.20.65.126:36935 was elected leader, sending a full tablet report...
I20240502 01:53:56.935106 20741 test_util.cc:268] Using random seed: -532137541
I20240502 01:53:56.948889 24028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49256:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:53:56.949728 24028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:53:56.974793 24271 tablet_service.cc:1467] Processing CreateTablet for tablet 1edcd38b8d8643ea896111553e49c16c (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 01:53:56.975328 24271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edcd38b8d8643ea896111553e49c16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.978708 24269 tablet_service.cc:1467] Processing CreateTablet for tablet 776becbedf8a4b89a59ea06cf8e2163a (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 01:53:56.979115 24269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776becbedf8a4b89a59ea06cf8e2163a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.982668 24271 tablet_service.cc:1467] Processing CreateTablet for tablet f0684f060716413796e90d5439d5d482 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 01:53:56.982853 24325 tablet_bootstrap.cc:492] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568: Bootstrap starting.
I20240502 01:53:56.982795 24268 tablet_service.cc:1467] Processing CreateTablet for tablet 92d3bd85ba5a4586b9f3ef23824c0287 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 01:53:56.983040 24271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0684f060716413796e90d5439d5d482. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.983218 24268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d3bd85ba5a4586b9f3ef23824c0287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.984786 24325 tablet_bootstrap.cc:654] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:56.985791 24262 tablet_service.cc:1467] Processing CreateTablet for tablet dc63402f38f94f50b33661f6d776f813 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 01:53:56.986186 24262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc63402f38f94f50b33661f6d776f813. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.988094 24271 tablet_service.cc:1467] Processing CreateTablet for tablet a64ccf8bf160452c98d7cf42884eadc5 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 01:53:56.988382 24271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a64ccf8bf160452c98d7cf42884eadc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.990801 24263 tablet_service.cc:1467] Processing CreateTablet for tablet 73d4415a5e104edcbfc621ab5604c431 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 01:53:56.990955 24267 tablet_service.cc:1467] Processing CreateTablet for tablet d516fd48dce443169e7e6011ea6a704d (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 01:53:56.991135 24263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d4415a5e104edcbfc621ab5604c431. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.991137 24325 tablet_bootstrap.cc:492] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568: No bootstrap required, opened a new log
I20240502 01:53:56.991323 24325 ts_tablet_manager.cc:1367] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240502 01:53:56.991351 24267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d516fd48dce443169e7e6011ea6a704d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.992033 24325 raft_consensus.cc:348] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } }
I20240502 01:53:56.992233 24325 raft_consensus.cc:374] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.992298 24325 raft_consensus.cc:729] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.992486 24325 consensus_queue.cc:260] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } }
I20240502 01:53:56.994868 24266 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf1c78d5f3f406d9029fe29deb6412a (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 01:53:56.995385 24266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf1c78d5f3f406d9029fe29deb6412a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.996013 24264 tablet_service.cc:1467] Processing CreateTablet for tablet 68d83f47ca7c4a5caeb4ffda5ebc9757 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 01:53:56.996372 24264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68d83f47ca7c4a5caeb4ffda5ebc9757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.997310 24265 tablet_service.cc:1467] Processing CreateTablet for tablet d3f5d1021b8b4932b7fe13b7fb933a9c (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 01:53:56.997704 24265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f5d1021b8b4932b7fe13b7fb933a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.000499 24325 ts_tablet_manager.cc:1398] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568: Time spent starting tablet: real 0.009s user 0.000s sys 0.008s
I20240502 01:53:57.000738 24325 tablet_bootstrap.cc:492] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568: Bootstrap starting.
I20240502 01:53:57.002426 24325 tablet_bootstrap.cc:654] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:56.978338 24270 tablet_service.cc:1467] Processing CreateTablet for tablet 947791f925a047cf82a9b28983f679bf (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 01:53:57.002854 24270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947791f925a047cf82a9b28983f679bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.007083 24196 tablet_service.cc:1467] Processing CreateTablet for tablet 1edcd38b8d8643ea896111553e49c16c (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 01:53:57.007513 24196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edcd38b8d8643ea896111553e49c16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.008757 24191 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf1c78d5f3f406d9029fe29deb6412a (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 01:53:57.009070 24325 tablet_bootstrap.cc:492] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568: No bootstrap required, opened a new log
I20240502 01:53:57.009181 24191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf1c78d5f3f406d9029fe29deb6412a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.009552 24187 tablet_service.cc:1467] Processing CreateTablet for tablet dc63402f38f94f50b33661f6d776f813 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 01:53:57.009972 24187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc63402f38f94f50b33661f6d776f813. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.010219 24121 tablet_service.cc:1467] Processing CreateTablet for tablet 1edcd38b8d8643ea896111553e49c16c (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 01:53:57.010598 24121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edcd38b8d8643ea896111553e49c16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.014793 24192 tablet_service.cc:1467] Processing CreateTablet for tablet d516fd48dce443169e7e6011ea6a704d (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 01:53:57.014925 24188 tablet_service.cc:1467] Processing CreateTablet for tablet 73d4415a5e104edcbfc621ab5604c431 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 01:53:57.015239 24192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d516fd48dce443169e7e6011ea6a704d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.015506 24188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d4415a5e104edcbfc621ab5604c431. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.015870 24193 tablet_service.cc:1467] Processing CreateTablet for tablet 92d3bd85ba5a4586b9f3ef23824c0287 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 01:53:57.016249 24193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d3bd85ba5a4586b9f3ef23824c0287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.017199 24120 tablet_service.cc:1467] Processing CreateTablet for tablet 947791f925a047cf82a9b28983f679bf (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 01:53:57.017619 24120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947791f925a047cf82a9b28983f679bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.018898 24189 tablet_service.cc:1467] Processing CreateTablet for tablet 68d83f47ca7c4a5caeb4ffda5ebc9757 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 01:53:57.019484 24189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68d83f47ca7c4a5caeb4ffda5ebc9757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.020037 24330 tablet_bootstrap.cc:492] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb: Bootstrap starting.
I20240502 01:53:57.021641 24330 tablet_bootstrap.cc:654] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.022779 24195 tablet_service.cc:1467] Processing CreateTablet for tablet 947791f925a047cf82a9b28983f679bf (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 01:53:57.023172 24195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947791f925a047cf82a9b28983f679bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.023083 24190 tablet_service.cc:1467] Processing CreateTablet for tablet d3f5d1021b8b4932b7fe13b7fb933a9c (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 01:53:57.023610 24190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f5d1021b8b4932b7fe13b7fb933a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.024845 24329 tablet_bootstrap.cc:492] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59: Bootstrap starting.
I20240502 01:53:57.025521 24192 tablet_service.cc:1467] Processing CreateTablet for tablet f0684f060716413796e90d5439d5d482 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 01:53:57.025844 24192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0684f060716413796e90d5439d5d482. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.026357 24329 tablet_bootstrap.cc:654] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.026777 24194 tablet_service.cc:1467] Processing CreateTablet for tablet 776becbedf8a4b89a59ea06cf8e2163a (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 01:53:57.027189 24194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776becbedf8a4b89a59ea06cf8e2163a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.027638 24196 tablet_service.cc:1467] Processing CreateTablet for tablet a64ccf8bf160452c98d7cf42884eadc5 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 01:53:57.028008 24196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a64ccf8bf160452c98d7cf42884eadc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.030243 24119 tablet_service.cc:1467] Processing CreateTablet for tablet 92d3bd85ba5a4586b9f3ef23824c0287 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 01:53:57.030730 24119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d3bd85ba5a4586b9f3ef23824c0287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.031291 24121 tablet_service.cc:1467] Processing CreateTablet for tablet 776becbedf8a4b89a59ea06cf8e2163a (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 01:53:57.031627 24121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776becbedf8a4b89a59ea06cf8e2163a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.034811 24118 tablet_service.cc:1467] Processing CreateTablet for tablet d516fd48dce443169e7e6011ea6a704d (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 01:53:57.035215 24118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d516fd48dce443169e7e6011ea6a704d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.035620 24117 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf1c78d5f3f406d9029fe29deb6412a (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 01:53:57.036038 24117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf1c78d5f3f406d9029fe29deb6412a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.009184 24325 ts_tablet_manager.cc:1367] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 01:53:57.035233 24329 tablet_bootstrap.cc:492] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59: No bootstrap required, opened a new log
I20240502 01:53:57.038143 24329 ts_tablet_manager.cc:1367] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 01:53:57.042883 24325 raft_consensus.cc:348] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.043036 24325 raft_consensus.cc:374] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.043113 24325 raft_consensus.cc:729] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.043917 24325 consensus_queue.cc:260] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.046324 24329 raft_consensus.cc:348] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } }
I20240502 01:53:57.046645 24329 raft_consensus.cc:374] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.046756 24329 raft_consensus.cc:729] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.047075 24325 ts_tablet_manager.cc:1398] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 01:53:57.047327 24325 tablet_bootstrap.cc:492] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568: Bootstrap starting.
I20240502 01:53:57.048126 24114 tablet_service.cc:1467] Processing CreateTablet for tablet 73d4415a5e104edcbfc621ab5604c431 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 01:53:57.048307 24112 tablet_service.cc:1467] Processing CreateTablet for tablet f0684f060716413796e90d5439d5d482 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 01:53:57.045162 24116 tablet_service.cc:1467] Processing CreateTablet for tablet d3f5d1021b8b4932b7fe13b7fb933a9c (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 01:53:57.050590 24113 tablet_service.cc:1467] Processing CreateTablet for tablet dc63402f38f94f50b33661f6d776f813 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 01:53:57.047089 24329 consensus_queue.cc:260] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } }
I20240502 01:53:57.053063 24120 tablet_service.cc:1467] Processing CreateTablet for tablet a64ccf8bf160452c98d7cf42884eadc5 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 01:53:57.053510 24120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a64ccf8bf160452c98d7cf42884eadc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.053844 24115 tablet_service.cc:1467] Processing CreateTablet for tablet 68d83f47ca7c4a5caeb4ffda5ebc9757 (DEFAULT_TABLE table=test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 01:53:57.056066 24114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d4415a5e104edcbfc621ab5604c431. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.059983 24112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0684f060716413796e90d5439d5d482. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.060189 24115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68d83f47ca7c4a5caeb4ffda5ebc9757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.061390 24113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc63402f38f94f50b33661f6d776f813. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.066552 24116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f5d1021b8b4932b7fe13b7fb933a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.079622 24325 tablet_bootstrap.cc:654] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.090368 24329 ts_tablet_manager.cc:1398] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59: Time spent starting tablet: real 0.052s user 0.005s sys 0.006s
I20240502 01:53:57.090737 24329 tablet_bootstrap.cc:492] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59: Bootstrap starting.
I20240502 01:53:57.092694 24330 tablet_bootstrap.cc:492] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb: No bootstrap required, opened a new log
I20240502 01:53:57.092804 24329 tablet_bootstrap.cc:654] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.092867 24330 ts_tablet_manager.cc:1367] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb: Time spent bootstrapping tablet: real 0.073s user 0.015s sys 0.030s
I20240502 01:53:57.096356 24330 raft_consensus.cc:348] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } }
I20240502 01:53:57.096792 24330 raft_consensus.cc:374] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.098512 24330 raft_consensus.cc:729] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.098847 24330 consensus_queue.cc:260] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } }
I20240502 01:53:57.099506 24325 tablet_bootstrap.cc:492] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568: No bootstrap required, opened a new log
I20240502 01:53:57.099689 24325 ts_tablet_manager.cc:1367] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.006s
I20240502 01:53:57.100248 24329 tablet_bootstrap.cc:492] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59: No bootstrap required, opened a new log
I20240502 01:53:57.100399 24329 ts_tablet_manager.cc:1367] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 01:53:57.100605 24325 raft_consensus.cc:348] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.100876 24325 raft_consensus.cc:374] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.100986 24325 raft_consensus.cc:729] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.100953 24329 raft_consensus.cc:348] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.101171 24329 raft_consensus.cc:374] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.101253 24329 raft_consensus.cc:729] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.101329 24325 consensus_queue.cc:260] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.101482 24329 consensus_queue.cc:260] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.102044 24325 ts_tablet_manager.cc:1398] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:57.102495 24329 ts_tablet_manager.cc:1398] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:57.102764 24329 tablet_bootstrap.cc:492] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59: Bootstrap starting.
I20240502 01:53:57.103606 24325 tablet_bootstrap.cc:492] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568: Bootstrap starting.
I20240502 01:53:57.104377 24329 tablet_bootstrap.cc:654] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.108083 24330 ts_tablet_manager.cc:1398] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb: Time spent starting tablet: real 0.015s user 0.005s sys 0.006s
I20240502 01:53:57.108134 24329 tablet_bootstrap.cc:492] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59: No bootstrap required, opened a new log
I20240502 01:53:57.108318 24329 ts_tablet_manager.cc:1367] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240502 01:53:57.108356 24330 tablet_bootstrap.cc:492] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb: Bootstrap starting.
I20240502 01:53:57.108974 24329 raft_consensus.cc:348] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.109148 24329 raft_consensus.cc:374] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.109242 24329 raft_consensus.cc:729] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.109493 24329 consensus_queue.cc:260] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.109920 24330 tablet_bootstrap.cc:654] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.110061 24329 ts_tablet_manager.cc:1398] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:57.110342 24329 tablet_bootstrap.cc:492] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59: Bootstrap starting.
I20240502 01:53:57.112025 24329 tablet_bootstrap.cc:654] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.112430 24325 tablet_bootstrap.cc:654] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.117820 24325 tablet_bootstrap.cc:492] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568: No bootstrap required, opened a new log
I20240502 01:53:57.118016 24325 ts_tablet_manager.cc:1367] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.006s
I20240502 01:53:57.118932 24325 raft_consensus.cc:348] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.119195 24325 raft_consensus.cc:374] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.119293 24325 raft_consensus.cc:729] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.119632 24325 consensus_queue.cc:260] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.123034 24325 ts_tablet_manager.cc:1398] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:53:57.125316 24325 tablet_bootstrap.cc:492] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568: Bootstrap starting.
I20240502 01:53:57.127560 24329 tablet_bootstrap.cc:492] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59: No bootstrap required, opened a new log
I20240502 01:53:57.127596 24325 tablet_bootstrap.cc:654] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.127732 24329 ts_tablet_manager.cc:1367] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.012s
I20240502 01:53:57.128585 24329 raft_consensus.cc:348] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.128798 24329 raft_consensus.cc:374] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.128883 24329 raft_consensus.cc:729] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.129155 24329 consensus_queue.cc:260] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.129787 24329 ts_tablet_manager.cc:1398] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:57.129859 24330 tablet_bootstrap.cc:492] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb: No bootstrap required, opened a new log
I20240502 01:53:57.130007 24330 ts_tablet_manager.cc:1367] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.008s
I20240502 01:53:57.130082 24329 tablet_bootstrap.cc:492] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59: Bootstrap starting.
I20240502 01:53:57.130223 24325 tablet_bootstrap.cc:492] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568: No bootstrap required, opened a new log
I20240502 01:53:57.130340 24325 ts_tablet_manager.cc:1367] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 01:53:57.130712 24330 raft_consensus.cc:348] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.130906 24330 raft_consensus.cc:374] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.130993 24330 raft_consensus.cc:729] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.130995 24325 raft_consensus.cc:348] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.131278 24325 raft_consensus.cc:374] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.131368 24325 raft_consensus.cc:729] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.131340 24330 consensus_queue.cc:260] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.131610 24325 consensus_queue.cc:260] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.131806 24329 tablet_bootstrap.cc:654] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.133739 24325 ts_tablet_manager.cc:1398] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:57.133853 24330 ts_tablet_manager.cc:1398] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:57.134030 24325 tablet_bootstrap.cc:492] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568: Bootstrap starting.
I20240502 01:53:57.134802 24330 tablet_bootstrap.cc:492] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb: Bootstrap starting.
I20240502 01:53:57.135685 24325 tablet_bootstrap.cc:654] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.136428 24330 tablet_bootstrap.cc:654] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.137794 24327 raft_consensus.cc:482] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.137984 24327 raft_consensus.cc:504] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.139254 24327 leader_election.cc:288] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:53:57.139717 24329 tablet_bootstrap.cc:492] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59: No bootstrap required, opened a new log
I20240502 01:53:57.139865 24329 ts_tablet_manager.cc:1367] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240502 01:53:57.140620 24329 raft_consensus.cc:348] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.140851 24329 raft_consensus.cc:374] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.140936 24329 raft_consensus.cc:729] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.141167 24333 raft_consensus.cc:482] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.141301 24333 raft_consensus.cc:504] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } }
I20240502 01:53:57.142102 24333 leader_election.cc:288] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253), 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551)
I20240502 01:53:57.156890 24329 consensus_queue.cc:260] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.159475 24329 ts_tablet_manager.cc:1398] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20240502 01:53:57.159849 24329 tablet_bootstrap.cc:492] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59: Bootstrap starting.
I20240502 01:53:57.162257 24329 tablet_bootstrap.cc:654] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.176201 24329 tablet_bootstrap.cc:492] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59: No bootstrap required, opened a new log
I20240502 01:53:57.176420 24329 ts_tablet_manager.cc:1367] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20240502 01:53:57.177259 24329 raft_consensus.cc:348] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.177567 24329 raft_consensus.cc:374] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.177714 24329 raft_consensus.cc:729] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.178059 24329 consensus_queue.cc:260] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.178679 24329 ts_tablet_manager.cc:1398] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:57.179109 24329 tablet_bootstrap.cc:492] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59: Bootstrap starting.
I20240502 01:53:57.180954 24329 tablet_bootstrap.cc:654] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.182511 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64ccf8bf160452c98d7cf42884eadc5" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb" is_pre_election: true
W20240502 01:53:57.183527 24246 leader_election.cc:341] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:57.183851 24333 raft_consensus.cc:482] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.184032 24333 raft_consensus.cc:504] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.184872 24333 leader_election.cc:288] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.186460 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edcd38b8d8643ea896111553e49c16c" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568" is_pre_election: true
I20240502 01:53:57.186806 24281 raft_consensus.cc:2454] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 0.
I20240502 01:53:57.187659 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568" is_pre_election: true
I20240502 01:53:57.187906 24281 raft_consensus.cc:2454] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 0.
I20240502 01:53:57.188390 24172 leader_election.cc:302] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9716594f476e436b91b93ceaed4bcf59, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:53:57.188702 24333 raft_consensus.cc:2781] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:57.188756 24172 leader_election.cc:302] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9716594f476e436b91b93ceaed4bcf59, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:53:57.188829 24333 raft_consensus.cc:482] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:57.189019 24333 raft_consensus.cc:3037] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.189322 24329 tablet_bootstrap.cc:492] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59: No bootstrap required, opened a new log
I20240502 01:53:57.189505 24329 ts_tablet_manager.cc:1367] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 01:53:57.190263 24325 tablet_bootstrap.cc:492] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568: No bootstrap required, opened a new log
I20240502 01:53:57.190167 24329 raft_consensus.cc:348] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.190443 24329 raft_consensus.cc:374] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.190493 24325 ts_tablet_manager.cc:1367] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.004s
I20240502 01:53:57.190608 24329 raft_consensus.cc:729] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.190861 24329 consensus_queue.cc:260] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.191408 24325 raft_consensus.cc:348] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.191669 24325 raft_consensus.cc:374] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.191785 24325 raft_consensus.cc:729] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.192103 24325 consensus_queue.cc:260] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.192898 24325 ts_tablet_manager.cc:1398] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:57.193260 24325 tablet_bootstrap.cc:492] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568: Bootstrap starting.
I20240502 01:53:57.195425 24325 tablet_bootstrap.cc:654] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.192391 24333 raft_consensus.cc:504] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } }
I20240502 01:53:57.198063 24130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb" is_pre_election: true
I20240502 01:53:57.198064 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edcd38b8d8643ea896111553e49c16c" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb" is_pre_election: true
I20240502 01:53:57.198666 24131 raft_consensus.cc:2454] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 0.
I20240502 01:53:57.198845 24333 leader_election.cc:288] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 election: Requested vote from peers a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253), 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551)
W20240502 01:53:57.199074 24171 leader_election.cc:341] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:57.199157 24330 tablet_bootstrap.cc:492] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb: No bootstrap required, opened a new log
I20240502 01:53:57.199337 24330 ts_tablet_manager.cc:1367] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb: Time spent bootstrapping tablet: real 0.065s user 0.012s sys 0.043s
I20240502 01:53:57.199484 24333 raft_consensus.cc:2781] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:57.199442 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edcd38b8d8643ea896111553e49c16c" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:53:57.199579 24333 raft_consensus.cc:482] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:57.199630 24131 raft_consensus.cc:3037] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.199689 24333 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.200258 24330 raft_consensus.cc:348] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.200541 24330 raft_consensus.cc:374] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.200641 24330 raft_consensus.cc:729] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.202387 24131 raft_consensus.cc:2454] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 1.
I20240502 01:53:57.202423 24333 raft_consensus.cc:504] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.202677 24330 consensus_queue.cc:260] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.202905 24171 leader_election.cc:302] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, 9716594f476e436b91b93ceaed4bcf59; no voters:
I20240502 01:53:57.203965 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:53:57.204187 24131 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.206331 24330 ts_tablet_manager.cc:1398] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20240502 01:53:57.207595 24330 tablet_bootstrap.cc:492] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb: Bootstrap starting.
I20240502 01:53:57.208086 24333 leader_election.cc:288] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.209029 24333 raft_consensus.cc:482] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.209218 24333 raft_consensus.cc:504] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.210320 24130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f5d1021b8b4932b7fe13b7fb933a9c" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb" is_pre_election: true
I20240502 01:53:57.210505 24131 raft_consensus.cc:2454] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 1.
W20240502 01:53:57.210944 24171 leader_election.cc:341] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:57.212522 24330 tablet_bootstrap.cc:654] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.213430 24333 leader_election.cc:288] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.213459 24171 leader_election.cc:302] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, 9716594f476e436b91b93ceaed4bcf59; no voters:
I20240502 01:53:57.213598 24333 raft_consensus.cc:2781] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:57.213740 24333 raft_consensus.cc:686] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Becoming Leader. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:53:57.214063 24333 consensus_queue.cc:237] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } }
I20240502 01:53:57.215131 24355 raft_consensus.cc:2781] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:57.215304 24355 raft_consensus.cc:686] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Becoming Leader. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:53:57.209213 24325 tablet_bootstrap.cc:492] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568: No bootstrap required, opened a new log
I20240502 01:53:57.215898 24325 ts_tablet_manager.cc:1367] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.003s
I20240502 01:53:57.215667 24355 consensus_queue.cc:237] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.216079 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edcd38b8d8643ea896111553e49c16c" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:53:57.216270 24281 raft_consensus.cc:3037] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.216873 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:53:57.217170 24280 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.219990 24281 raft_consensus.cc:2454] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 1.
I20240502 01:53:57.221149 24329 ts_tablet_manager.cc:1398] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59: Time spent starting tablet: real 0.032s user 0.004s sys 0.015s
I20240502 01:53:57.221148 24280 raft_consensus.cc:2454] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 1.
I20240502 01:53:57.221498 24329 tablet_bootstrap.cc:492] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59: Bootstrap starting.
I20240502 01:53:57.221838 24325 raft_consensus.cc:348] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.222082 24325 raft_consensus.cc:374] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.222218 24325 raft_consensus.cc:729] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.222582 24325 consensus_queue.cc:260] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.223235 24329 tablet_bootstrap.cc:654] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.223675 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f5d1021b8b4932b7fe13b7fb933a9c" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568" is_pre_election: true
I20240502 01:53:57.224521 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64ccf8bf160452c98d7cf42884eadc5" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59" is_pre_election: true
I20240502 01:53:57.224786 24207 raft_consensus.cc:2454] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 0.
W20240502 01:53:57.225281 24172 leader_election.cc:341] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:57.225440 24172 leader_election.cc:302] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9716594f476e436b91b93ceaed4bcf59; no voters: 227af58cb9554c438ae592baff6af4fb, a6f86494e4de4d3c851c5dab07b40568
I20240502 01:53:57.225988 24246 leader_election.cc:302] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9716594f476e436b91b93ceaed4bcf59, a6f86494e4de4d3c851c5dab07b40568; no voters: 227af58cb9554c438ae592baff6af4fb
I20240502 01:53:57.226390 24327 raft_consensus.cc:2781] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:57.226543 24327 raft_consensus.cc:482] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:57.226577 24333 raft_consensus.cc:2726] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:57.226660 24327 raft_consensus.cc:3037] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.227975 24330 tablet_bootstrap.cc:492] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb: No bootstrap required, opened a new log
I20240502 01:53:57.228129 24330 ts_tablet_manager.cc:1367] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.007s
I20240502 01:53:57.228700 24330 raft_consensus.cc:348] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.228843 24330 raft_consensus.cc:374] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.228897 24330 raft_consensus.cc:729] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.229081 24330 consensus_queue.cc:260] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.229512 24330 ts_tablet_manager.cc:1398] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:53:57.229808 24330 tablet_bootstrap.cc:492] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb: Bootstrap starting.
I20240502 01:53:57.230643 24327 raft_consensus.cc:504] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.231426 24330 tablet_bootstrap.cc:654] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.231869 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64ccf8bf160452c98d7cf42884eadc5" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb"
W20240502 01:53:57.232872 24246 leader_election.cc:341] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:57.232684 24027 catalog_manager.cc:5484] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66). New cstate: current_term: 1 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:57.233584 24027 catalog_manager.cc:5484] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66). New cstate: current_term: 1 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.236920 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64ccf8bf160452c98d7cf42884eadc5" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:53:57.237016 24327 leader_election.cc:288] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:53:57.237103 24207 raft_consensus.cc:3037] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.226150 24325 ts_tablet_manager.cc:1398] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 01:53:57.239166 24329 tablet_bootstrap.cc:492] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59: No bootstrap required, opened a new log
I20240502 01:53:57.239332 24329 ts_tablet_manager.cc:1367] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.011s
I20240502 01:53:57.239697 24330 tablet_bootstrap.cc:492] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb: No bootstrap required, opened a new log
I20240502 01:53:57.239812 24325 tablet_bootstrap.cc:492] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568: Bootstrap starting.
I20240502 01:53:57.239850 24330 ts_tablet_manager.cc:1367] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240502 01:53:57.240089 24329 raft_consensus.cc:348] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.240304 24329 raft_consensus.cc:374] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.240398 24329 raft_consensus.cc:729] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.240504 24207 raft_consensus.cc:2454] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 1.
I20240502 01:53:57.240638 24329 consensus_queue.cc:260] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.240926 24246 leader_election.cc:302] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9716594f476e436b91b93ceaed4bcf59, a6f86494e4de4d3c851c5dab07b40568; no voters: 227af58cb9554c438ae592baff6af4fb
I20240502 01:53:57.241232 24329 ts_tablet_manager.cc:1398] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:57.241376 24327 raft_consensus.cc:2781] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:57.241530 24329 tablet_bootstrap.cc:492] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59: Bootstrap starting.
I20240502 01:53:57.240643 24330 raft_consensus.cc:348] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.241861 24330 raft_consensus.cc:374] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.241962 24330 raft_consensus.cc:729] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.242208 24325 tablet_bootstrap.cc:654] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.242254 24330 consensus_queue.cc:260] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.243202 24329 tablet_bootstrap.cc:654] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.245334 24330 ts_tablet_manager.cc:1398] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:53:57.245575 24329 tablet_bootstrap.cc:492] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59: No bootstrap required, opened a new log
I20240502 01:53:57.245621 24330 tablet_bootstrap.cc:492] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb: Bootstrap starting.
I20240502 01:53:57.245745 24329 ts_tablet_manager.cc:1367] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:57.246887 24329 raft_consensus.cc:348] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.247143 24329 raft_consensus.cc:374] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.247257 24329 raft_consensus.cc:729] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.247535 24329 consensus_queue.cc:260] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.248227 24329 ts_tablet_manager.cc:1398] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:57.248561 24329 tablet_bootstrap.cc:492] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59: Bootstrap starting.
I20240502 01:53:57.248612 24355 raft_consensus.cc:482] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.248769 24355 raft_consensus.cc:504] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.249477 24355 leader_election.cc:288] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.250149 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d4415a5e104edcbfc621ab5604c431" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb" is_pre_election: true
I20240502 01:53:57.247175 24330 tablet_bootstrap.cc:654] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.250540 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d4415a5e104edcbfc621ab5604c431" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568" is_pre_election: true
W20240502 01:53:57.250671 24171 leader_election.cc:341] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:57.250787 24280 raft_consensus.cc:2454] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 0.
I20240502 01:53:57.251216 24325 tablet_bootstrap.cc:492] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568: No bootstrap required, opened a new log
I20240502 01:53:57.251215 24172 leader_election.cc:302] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9716594f476e436b91b93ceaed4bcf59, a6f86494e4de4d3c851c5dab07b40568; no voters: 227af58cb9554c438ae592baff6af4fb
I20240502 01:53:57.251421 24329 tablet_bootstrap.cc:654] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.251606 24355 raft_consensus.cc:2781] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:57.251731 24355 raft_consensus.cc:482] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:57.251854 24355 raft_consensus.cc:3037] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.252004 24325 ts_tablet_manager.cc:1367] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20240502 01:53:57.252872 24325 raft_consensus.cc:348] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.253130 24325 raft_consensus.cc:374] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.253266 24325 raft_consensus.cc:729] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.253571 24325 consensus_queue.cc:260] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.253801 24330 tablet_bootstrap.cc:492] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb: No bootstrap required, opened a new log
I20240502 01:53:57.253947 24330 ts_tablet_manager.cc:1367] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:53:57.254820 24330 raft_consensus.cc:348] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.255065 24330 raft_consensus.cc:374] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.255158 24330 raft_consensus.cc:729] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.255492 24355 raft_consensus.cc:504] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.256438 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d4415a5e104edcbfc621ab5604c431" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:53:57.256570 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d4415a5e104edcbfc621ab5604c431" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:53:57.256449 24336 raft_consensus.cc:482] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.256784 24355 leader_election.cc:288] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.255493 24330 consensus_queue.cc:260] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.256882 24336 raft_consensus.cc:504] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.258147 24336 leader_election.cc:288] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.260339 24131 raft_consensus.cc:3037] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.256753 24329 tablet_bootstrap.cc:492] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59: No bootstrap required, opened a new log
I20240502 01:53:57.260756 24329 ts_tablet_manager.cc:1367] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20240502 01:53:57.262492 24325 ts_tablet_manager.cc:1398] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568: Time spent starting tablet: real 0.010s user 0.002s sys 0.007s
I20240502 01:53:57.261632 24329 raft_consensus.cc:348] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.263156 24329 raft_consensus.cc:374] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.263437 24329 raft_consensus.cc:729] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.263317 24325 tablet_bootstrap.cc:492] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568: Bootstrap starting.
I20240502 01:53:57.263679 24329 consensus_queue.cc:260] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.264173 24329 ts_tablet_manager.cc:1398] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:53:57.256721 24280 raft_consensus.cc:3037] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.265712 24325 tablet_bootstrap.cc:654] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.266615 24367 raft_consensus.cc:482] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.267369 24367 raft_consensus.cc:504] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.269282 24367 leader_election.cc:288] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:53:57.270063 24280 raft_consensus.cc:2454] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 1.
I20240502 01:53:57.273532 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc63402f38f94f50b33661f6d776f813" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59" is_pre_election: true
I20240502 01:53:57.273808 24207 raft_consensus.cc:2454] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 0.
I20240502 01:53:57.274178 24329 tablet_bootstrap.cc:492] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59: Bootstrap starting.
I20240502 01:53:57.274837 24246 leader_election.cc:302] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9716594f476e436b91b93ceaed4bcf59, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:53:57.275015 24131 raft_consensus.cc:2454] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 1.
I20240502 01:53:57.275674 24171 leader_election.cc:302] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, 9716594f476e436b91b93ceaed4bcf59; no voters:
I20240502 01:53:57.275970 24355 raft_consensus.cc:2781] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:57.276012 24329 tablet_bootstrap.cc:654] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.276191 24355 raft_consensus.cc:686] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Becoming Leader. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:53:57.276517 24355 consensus_queue.cc:237] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.277688 24367 raft_consensus.cc:2781] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:57.277983 24367 raft_consensus.cc:482] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:57.278174 24367 raft_consensus.cc:3037] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.279546 24327 raft_consensus.cc:686] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Becoming Leader. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Running, Role: LEADER
I20240502 01:53:57.279896 24327 consensus_queue.cc:237] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.282686 24367 raft_consensus.cc:504] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.284173 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc63402f38f94f50b33661f6d776f813" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:53:57.284426 24207 raft_consensus.cc:3037] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.288106 24207 raft_consensus.cc:2454] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 1.
I20240502 01:53:57.288686 24246 leader_election.cc:302] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9716594f476e436b91b93ceaed4bcf59, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:53:57.289016 24362 raft_consensus.cc:2781] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:57.289167 24367 leader_election.cc:288] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:53:57.289206 24362 raft_consensus.cc:686] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Becoming Leader. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Running, Role: LEADER
I20240502 01:53:57.289611 24362 consensus_queue.cc:237] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.315627 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc63402f38f94f50b33661f6d776f813" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb" is_pre_election: true
I20240502 01:53:57.315778 24333 raft_consensus.cc:482] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.316016 24333 raft_consensus.cc:504] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.316360 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc63402f38f94f50b33661f6d776f813" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb"
W20240502 01:53:57.316840 24246 leader_election.cc:341] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:57.316917 24333 leader_election.cc:288] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
W20240502 01:53:57.317390 24246 leader_election.cc:341] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:57.318395 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0684f060716413796e90d5439d5d482" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb" is_pre_election: true
I20240502 01:53:57.318679 24131 raft_consensus.cc:2454] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 0.
I20240502 01:53:57.319145 24171 leader_election.cc:302] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, 9716594f476e436b91b93ceaed4bcf59; no voters:
I20240502 01:53:57.319502 24333 raft_consensus.cc:2781] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:57.319633 24333 raft_consensus.cc:482] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:57.319746 24333 raft_consensus.cc:3037] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.323761 24333 raft_consensus.cc:504] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.325157 24333 leader_election.cc:288] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.325479 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0684f060716413796e90d5439d5d482" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:53:57.325690 24131 raft_consensus.cc:3037] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.329578 24131 raft_consensus.cc:2454] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 1.
I20240502 01:53:57.330005 24171 leader_election.cc:302] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, 9716594f476e436b91b93ceaed4bcf59; no voters:
I20240502 01:53:57.330449 24367 raft_consensus.cc:482] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.330713 24367 raft_consensus.cc:504] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.331642 24367 leader_election.cc:288] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:53:57.330485 24333 raft_consensus.cc:2781] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:57.332376 24333 raft_consensus.cc:686] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Becoming Leader. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:53:57.332541 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0684f060716413796e90d5439d5d482" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59" is_pre_election: true
I20240502 01:53:57.332898 24333 consensus_queue.cc:237] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.333652 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0684f060716413796e90d5439d5d482" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb" is_pre_election: true
I20240502 01:53:57.334009 24131 raft_consensus.cc:2379] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a6f86494e4de4d3c851c5dab07b40568 in current term 1: Already voted for candidate 9716594f476e436b91b93ceaed4bcf59 in this term.
I20240502 01:53:57.334676 24246 leader_election.cc:302] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6f86494e4de4d3c851c5dab07b40568; no voters: 227af58cb9554c438ae592baff6af4fb, 9716594f476e436b91b93ceaed4bcf59
I20240502 01:53:57.334993 24362 raft_consensus.cc:3037] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.340148 24026 catalog_manager.cc:5484] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67). New cstate: current_term: 1 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:57.340907 24027 catalog_manager.cc:5484] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66). New cstate: current_term: 1 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.341449 24027 catalog_manager.cc:5484] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66). New cstate: current_term: 1 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.345906 24027 catalog_manager.cc:5484] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67). New cstate: current_term: 1 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:57.347456 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0684f060716413796e90d5439d5d482" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:53:57.348289 24362 raft_consensus.cc:2726] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:57.349314 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0684f060716413796e90d5439d5d482" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568" is_pre_election: true
I20240502 01:53:57.350576 24333 raft_consensus.cc:482] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.350740 24333 raft_consensus.cc:504] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.351341 24281 raft_consensus.cc:2454] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 1.
I20240502 01:53:57.351737 24333 leader_election.cc:288] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.352231 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d83f47ca7c4a5caeb4ffda5ebc9757" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb" is_pre_election: true
W20240502 01:53:57.352712 24171 leader_election.cc:341] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:57.354588 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d83f47ca7c4a5caeb4ffda5ebc9757" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568" is_pre_election: true
I20240502 01:53:57.354765 24280 raft_consensus.cc:2454] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 0.
I20240502 01:53:57.354352 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776becbedf8a4b89a59ea06cf8e2163a" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568" is_pre_election: true
I20240502 01:53:57.355100 24172 leader_election.cc:302] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9716594f476e436b91b93ceaed4bcf59, a6f86494e4de4d3c851c5dab07b40568; no voters: 227af58cb9554c438ae592baff6af4fb
W20240502 01:53:57.355371 24096 leader_election.cc:341] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:57.355581 24355 raft_consensus.cc:2781] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:57.355683 24355 raft_consensus.cc:482] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:57.355783 24355 raft_consensus.cc:3037] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.357627 24355 raft_consensus.cc:504] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.358300 24355 leader_election.cc:288] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.358585 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d83f47ca7c4a5caeb4ffda5ebc9757" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb"
W20240502 01:53:57.359098 24171 leader_election.cc:341] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:57.361351 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d83f47ca7c4a5caeb4ffda5ebc9757" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:53:57.361598 24281 raft_consensus.cc:3037] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.362290 24325 tablet_bootstrap.cc:492] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568: No bootstrap required, opened a new log
I20240502 01:53:57.362427 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776becbedf8a4b89a59ea06cf8e2163a" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59" is_pre_election: true
I20240502 01:53:57.362556 24325 ts_tablet_manager.cc:1367] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568: Time spent bootstrapping tablet: real 0.099s user 0.017s sys 0.052s
W20240502 01:53:57.363006 24095 leader_election.cc:341] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:57.363062 24329 tablet_bootstrap.cc:492] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59: No bootstrap required, opened a new log
I20240502 01:53:57.363184 24095 leader_election.cc:302] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb; no voters: 9716594f476e436b91b93ceaed4bcf59, a6f86494e4de4d3c851c5dab07b40568
I20240502 01:53:57.363253 24329 ts_tablet_manager.cc:1367] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59: Time spent bootstrapping tablet: real 0.089s user 0.008s sys 0.008s
I20240502 01:53:57.363723 24336 raft_consensus.cc:2726] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:57.364183 24329 raft_consensus.cc:348] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.364394 24330 ts_tablet_manager.cc:1398] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb: Time spent starting tablet: real 0.110s user 0.034s sys 0.069s
I20240502 01:53:57.364416 24329 raft_consensus.cc:374] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.364547 24329 raft_consensus.cc:729] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.364775 24325 raft_consensus.cc:348] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.364802 24329 consensus_queue.cc:260] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.364979 24325 raft_consensus.cc:374] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.365087 24325 raft_consensus.cc:729] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.365303 24329 ts_tablet_manager.cc:1398] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:57.365382 24325 consensus_queue.cc:260] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.365610 24329 tablet_bootstrap.cc:492] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59: Bootstrap starting.
I20240502 01:53:57.365926 24325 ts_tablet_manager.cc:1398] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:57.367004 24330 tablet_bootstrap.cc:492] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb: Bootstrap starting.
I20240502 01:53:57.367017 24325 tablet_bootstrap.cc:492] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568: Bootstrap starting.
I20240502 01:53:57.367622 24329 tablet_bootstrap.cc:654] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.368888 24325 tablet_bootstrap.cc:654] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.368888 24330 tablet_bootstrap.cc:654] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.374922 24281 raft_consensus.cc:2454] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 1.
I20240502 01:53:57.375448 24172 leader_election.cc:302] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9716594f476e436b91b93ceaed4bcf59, a6f86494e4de4d3c851c5dab07b40568; no voters: 227af58cb9554c438ae592baff6af4fb
I20240502 01:53:57.375773 24355 raft_consensus.cc:2781] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:57.375916 24355 raft_consensus.cc:686] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Becoming Leader. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:53:57.376240 24355 consensus_queue.cc:237] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.377410 24325 tablet_bootstrap.cc:492] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568: No bootstrap required, opened a new log
I20240502 01:53:57.377593 24325 ts_tablet_manager.cc:1367] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240502 01:53:57.377677 24330 tablet_bootstrap.cc:492] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb: No bootstrap required, opened a new log
I20240502 01:53:57.377828 24330 ts_tablet_manager.cc:1367] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20240502 01:53:57.378383 24325 raft_consensus.cc:348] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.378522 24330 raft_consensus.cc:348] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.378685 24325 raft_consensus.cc:374] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.378717 24330 raft_consensus.cc:374] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.378772 24325 raft_consensus.cc:729] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.378798 24330 raft_consensus.cc:729] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.379031 24330 consensus_queue.cc:260] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.379041 24325 consensus_queue.cc:260] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.379257 24329 tablet_bootstrap.cc:492] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59: No bootstrap required, opened a new log
I20240502 01:53:57.379428 24329 ts_tablet_manager.cc:1367] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20240502 01:53:57.379593 24330 ts_tablet_manager.cc:1398] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:57.379395 24026 catalog_manager.cc:5484] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66). New cstate: current_term: 1 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.380089 24329 raft_consensus.cc:348] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.380307 24329 raft_consensus.cc:374] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.380416 24329 raft_consensus.cc:729] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.380635 24329 consensus_queue.cc:260] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.381194 24329 ts_tablet_manager.cc:1398] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:57.382169 24325 ts_tablet_manager.cc:1398] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:53:57.382516 24325 tablet_bootstrap.cc:492] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568: Bootstrap starting.
I20240502 01:53:57.382563 24330 tablet_bootstrap.cc:492] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb: Bootstrap starting.
I20240502 01:53:57.384368 24325 tablet_bootstrap.cc:654] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.384666 24330 tablet_bootstrap.cc:654] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.387027 24330 tablet_bootstrap.cc:492] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb: No bootstrap required, opened a new log
I20240502 01:53:57.387264 24330 ts_tablet_manager.cc:1367] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20240502 01:53:57.388337 24330 raft_consensus.cc:348] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.388764 24330 raft_consensus.cc:374] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.388893 24330 raft_consensus.cc:729] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.389212 24330 consensus_queue.cc:260] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.390097 24330 ts_tablet_manager.cc:1398] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240502 01:53:57.390512 24330 tablet_bootstrap.cc:492] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb: Bootstrap starting.
I20240502 01:53:57.392611 24330 tablet_bootstrap.cc:654] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.392717 24325 tablet_bootstrap.cc:492] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568: No bootstrap required, opened a new log
I20240502 01:53:57.392866 24325 ts_tablet_manager.cc:1367] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240502 01:53:57.393698 24325 raft_consensus.cc:348] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.393858 24325 raft_consensus.cc:374] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.393949 24325 raft_consensus.cc:729] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.394184 24325 consensus_queue.cc:260] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.394517 24330 tablet_bootstrap.cc:492] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb: No bootstrap required, opened a new log
I20240502 01:53:57.394659 24330 ts_tablet_manager.cc:1367] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:53:57.394759 24325 ts_tablet_manager.cc:1398] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:57.395053 24325 tablet_bootstrap.cc:492] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568: Bootstrap starting.
I20240502 01:53:57.395380 24330 raft_consensus.cc:348] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.395591 24330 raft_consensus.cc:374] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.395715 24330 raft_consensus.cc:729] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.396001 24330 consensus_queue.cc:260] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.396567 24330 ts_tablet_manager.cc:1398] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:57.396783 24325 tablet_bootstrap.cc:654] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.396874 24330 tablet_bootstrap.cc:492] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb: Bootstrap starting.
I20240502 01:53:57.398977 24330 tablet_bootstrap.cc:654] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.401309 24330 tablet_bootstrap.cc:492] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb: No bootstrap required, opened a new log
I20240502 01:53:57.401516 24330 ts_tablet_manager.cc:1367] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:53:57.402462 24330 raft_consensus.cc:348] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.402737 24330 raft_consensus.cc:374] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.402827 24330 raft_consensus.cc:729] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.403048 24330 consensus_queue.cc:260] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.403542 24330 ts_tablet_manager.cc:1398] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:57.403549 24325 tablet_bootstrap.cc:492] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568: No bootstrap required, opened a new log
I20240502 01:53:57.403725 24325 ts_tablet_manager.cc:1367] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.005s
I20240502 01:53:57.403932 24330 tablet_bootstrap.cc:492] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb: Bootstrap starting.
I20240502 01:53:57.404398 24325 raft_consensus.cc:348] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.404709 24325 raft_consensus.cc:374] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.404812 24325 raft_consensus.cc:729] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.405129 24325 consensus_queue.cc:260] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.405588 24330 tablet_bootstrap.cc:654] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.405645 24325 ts_tablet_manager.cc:1398] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:57.408991 24330 tablet_bootstrap.cc:492] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb: No bootstrap required, opened a new log
I20240502 01:53:57.409161 24330 ts_tablet_manager.cc:1367] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.003s
I20240502 01:53:57.409869 24330 raft_consensus.cc:348] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.410009 24330 raft_consensus.cc:374] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.410071 24330 raft_consensus.cc:729] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.410218 24330 consensus_queue.cc:260] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.410640 24330 ts_tablet_manager.cc:1398] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:53:57.410887 24330 tablet_bootstrap.cc:492] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb: Bootstrap starting.
I20240502 01:53:57.412195 24330 tablet_bootstrap.cc:654] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.413941 24330 tablet_bootstrap.cc:492] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb: No bootstrap required, opened a new log
I20240502 01:53:57.414086 24330 ts_tablet_manager.cc:1367] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 01:53:57.414619 24330 raft_consensus.cc:348] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.414810 24330 raft_consensus.cc:374] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.414891 24330 raft_consensus.cc:729] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.415103 24330 consensus_queue.cc:260] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.415551 24330 ts_tablet_manager.cc:1398] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:53:57.431201 24333 raft_consensus.cc:482] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.431373 24333 raft_consensus.cc:504] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.431989 24333 leader_election.cc:288] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.432413 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d516fd48dce443169e7e6011ea6a704d" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb" is_pre_election: true
I20240502 01:53:57.432484 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d516fd48dce443169e7e6011ea6a704d" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568" is_pre_election: true
I20240502 01:53:57.432628 24131 raft_consensus.cc:2454] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 0.
I20240502 01:53:57.432713 24281 raft_consensus.cc:2454] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 0.
I20240502 01:53:57.433013 24171 leader_election.cc:302] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, 9716594f476e436b91b93ceaed4bcf59; no voters:
I20240502 01:53:57.433306 24333 raft_consensus.cc:2781] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:57.433395 24333 raft_consensus.cc:482] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:57.433473 24333 raft_consensus.cc:3037] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.435429 24333 raft_consensus.cc:504] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.436089 24333 leader_election.cc:288] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.436479 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d516fd48dce443169e7e6011ea6a704d" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:53:57.436548 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d516fd48dce443169e7e6011ea6a704d" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:53:57.436700 24131 raft_consensus.cc:3037] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.436764 24281 raft_consensus.cc:3037] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.438680 24281 raft_consensus.cc:2454] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 1.
I20240502 01:53:57.438741 24131 raft_consensus.cc:2454] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 1.
I20240502 01:53:57.439060 24172 leader_election.cc:302] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9716594f476e436b91b93ceaed4bcf59, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:53:57.439337 24333 raft_consensus.cc:2781] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:57.439497 24333 raft_consensus.cc:686] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Becoming Leader. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:53:57.439754 24333 consensus_queue.cc:237] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.442086 24026 catalog_manager.cc:5484] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66). New cstate: current_term: 1 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.520182 24368 raft_consensus.cc:482] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.520450 24368 raft_consensus.cc:504] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.521462 24368 leader_election.cc:288] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.521937 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947791f925a047cf82a9b28983f679bf" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59" is_pre_election: true
I20240502 01:53:57.521960 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947791f925a047cf82a9b28983f679bf" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568" is_pre_election: true
I20240502 01:53:57.522238 24207 raft_consensus.cc:2454] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 0.
I20240502 01:53:57.522807 24095 leader_election.cc:302] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, 9716594f476e436b91b93ceaed4bcf59; no voters:
I20240502 01:53:57.523087 24368 raft_consensus.cc:2781] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:57.523205 24368 raft_consensus.cc:482] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:57.523296 24368 raft_consensus.cc:3037] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.525365 24336 raft_consensus.cc:482] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.525274 24368 raft_consensus.cc:504] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.525468 24336 raft_consensus.cc:504] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.525935 24336 leader_election.cc:288] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.526319 24368 leader_election.cc:288] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 election: Requested vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.526552 24206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc63402f38f94f50b33661f6d776f813" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59" is_pre_election: true
I20240502 01:53:57.526557 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947791f925a047cf82a9b28983f679bf" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:53:57.526706 24279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc63402f38f94f50b33661f6d776f813" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568" is_pre_election: true
I20240502 01:53:57.526798 24206 raft_consensus.cc:2379] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 227af58cb9554c438ae592baff6af4fb in current term 1: Already voted for candidate a6f86494e4de4d3c851c5dab07b40568 in this term.
I20240502 01:53:57.526801 24280 raft_consensus.cc:3037] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.529522 24280 raft_consensus.cc:2454] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 1.
I20240502 01:53:57.529806 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947791f925a047cf82a9b28983f679bf" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:53:57.529870 24095 leader_election.cc:302] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb; no voters: 9716594f476e436b91b93ceaed4bcf59, a6f86494e4de4d3c851c5dab07b40568
I20240502 01:53:57.529987 24207 raft_consensus.cc:3037] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.529963 24096 leader_election.cc:302] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:53:57.530211 24336 raft_consensus.cc:3037] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.530303 24368 raft_consensus.cc:2781] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:57.531208 24281 raft_consensus.cc:2362] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 1. Re-sending same reply.
I20240502 01:53:57.531886 24368 raft_consensus.cc:686] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Becoming Leader. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:53:57.532404 24368 consensus_queue.cc:237] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.532541 24336 raft_consensus.cc:2726] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:57.532940 24207 raft_consensus.cc:2454] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 1.
I20240502 01:53:57.535188 24026 catalog_manager.cc:5484] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 227af58cb9554c438ae592baff6af4fb (127.20.65.65). New cstate: current_term: 1 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.550699 24362 raft_consensus.cc:482] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.550828 24362 raft_consensus.cc:504] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.551241 24362 leader_election.cc:288] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:53:57.551687 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776becbedf8a4b89a59ea06cf8e2163a" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb" is_pre_election: true
I20240502 01:53:57.551803 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776becbedf8a4b89a59ea06cf8e2163a" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59" is_pre_election: true
I20240502 01:53:57.551872 24131 raft_consensus.cc:2454] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 0.
I20240502 01:53:57.551982 24207 raft_consensus.cc:2454] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 0.
I20240502 01:53:57.552242 24246 leader_election.cc:302] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:53:57.552505 24362 raft_consensus.cc:2781] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:57.552572 24362 raft_consensus.cc:482] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:57.552619 24362 raft_consensus.cc:3037] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.554422 24362 raft_consensus.cc:504] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.554898 24362 leader_election.cc:288] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:53:57.555312 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776becbedf8a4b89a59ea06cf8e2163a" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:53:57.555467 24131 raft_consensus.cc:3037] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.555497 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776becbedf8a4b89a59ea06cf8e2163a" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:53:57.555660 24207 raft_consensus.cc:3037] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.557283 24131 raft_consensus.cc:2454] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 1.
I20240502 01:53:57.557377 24207 raft_consensus.cc:2454] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 1.
I20240502 01:53:57.557667 24246 leader_election.cc:302] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:53:57.557885 24362 raft_consensus.cc:2781] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:57.558003 24362 raft_consensus.cc:686] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Becoming Leader. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Running, Role: LEADER
I20240502 01:53:57.558215 24362 consensus_queue.cc:237] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.560643 24026 catalog_manager.cc:5484] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67). New cstate: current_term: 1 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:57.577574 24368 raft_consensus.cc:482] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.577746 24368 raft_consensus.cc:504] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.578286 24368 leader_election.cc:288] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.578781 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d83f47ca7c4a5caeb4ffda5ebc9757" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59" is_pre_election: true
I20240502 01:53:57.578781 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d83f47ca7c4a5caeb4ffda5ebc9757" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568" is_pre_election: true
I20240502 01:53:57.579066 24281 raft_consensus.cc:2379] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 227af58cb9554c438ae592baff6af4fb in current term 1: Already voted for candidate 9716594f476e436b91b93ceaed4bcf59 in this term.
I20240502 01:53:57.579423 24096 leader_election.cc:302] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb; no voters: 9716594f476e436b91b93ceaed4bcf59, a6f86494e4de4d3c851c5dab07b40568
I20240502 01:53:57.579641 24368 raft_consensus.cc:3037] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.580783 24336 raft_consensus.cc:482] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.580922 24336 raft_consensus.cc:504] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.581518 24336 leader_election.cc:288] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.581893 24368 raft_consensus.cc:2726] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:57.581854 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64ccf8bf160452c98d7cf42884eadc5" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59" is_pre_election: true
I20240502 01:53:57.582059 24207 raft_consensus.cc:2379] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 227af58cb9554c438ae592baff6af4fb in current term 1: Already voted for candidate a6f86494e4de4d3c851c5dab07b40568 in this term.
I20240502 01:53:57.582012 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64ccf8bf160452c98d7cf42884eadc5" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568" is_pre_election: true
I20240502 01:53:57.582394 24095 leader_election.cc:302] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb; no voters: 9716594f476e436b91b93ceaed4bcf59, a6f86494e4de4d3c851c5dab07b40568
I20240502 01:53:57.582690 24368 raft_consensus.cc:3037] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.584285 24368 raft_consensus.cc:2726] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:57.602993 24368 raft_consensus.cc:482] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.603166 24368 raft_consensus.cc:504] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.603724 24368 leader_election.cc:288] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.604199 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568" is_pre_election: true
I20240502 01:53:57.604199 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59" is_pre_election: true
I20240502 01:53:57.604416 24207 raft_consensus.cc:2454] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 0.
I20240502 01:53:57.604416 24281 raft_consensus.cc:2454] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 0.
I20240502 01:53:57.604770 24095 leader_election.cc:302] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, 9716594f476e436b91b93ceaed4bcf59; no voters:
I20240502 01:53:57.605021 24368 raft_consensus.cc:2781] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:57.605101 24368 raft_consensus.cc:482] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:57.605175 24368 raft_consensus.cc:3037] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.607326 24368 raft_consensus.cc:504] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.607937 24368 leader_election.cc:288] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 election: Requested vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.608309 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:53:57.608309 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:53:57.608513 24207 raft_consensus.cc:3037] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.608513 24281 raft_consensus.cc:3037] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.611229 24207 raft_consensus.cc:2454] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 1.
I20240502 01:53:57.611229 24281 raft_consensus.cc:2454] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 1.
I20240502 01:53:57.611598 24096 leader_election.cc:302] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:53:57.611832 24368 raft_consensus.cc:2781] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:57.611939 24368 raft_consensus.cc:686] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Becoming Leader. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:53:57.612174 24368 consensus_queue.cc:237] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.614538 24026 catalog_manager.cc:5484] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 227af58cb9554c438ae592baff6af4fb (127.20.65.65). New cstate: current_term: 1 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.617931 24333 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.622396 24354 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.645190 24368 raft_consensus.cc:482] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.645344 24368 raft_consensus.cc:504] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.645867 24368 leader_election.cc:288] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.646443 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f5d1021b8b4932b7fe13b7fb933a9c" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568" is_pre_election: true
I20240502 01:53:57.646332 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f5d1021b8b4932b7fe13b7fb933a9c" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59" is_pre_election: true
I20240502 01:53:57.646654 24281 raft_consensus.cc:2454] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 0.
I20240502 01:53:57.646684 24207 raft_consensus.cc:2454] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 0.
I20240502 01:53:57.646987 24096 leader_election.cc:302] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:53:57.647212 24368 raft_consensus.cc:2781] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:57.647282 24368 raft_consensus.cc:482] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:57.647336 24368 raft_consensus.cc:3037] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.649178 24354 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.649906 24368 raft_consensus.cc:504] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.650563 24368 leader_election.cc:288] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 election: Requested vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:57.650866 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f5d1021b8b4932b7fe13b7fb933a9c" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:53:57.651018 24207 raft_consensus.cc:3037] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.651219 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f5d1021b8b4932b7fe13b7fb933a9c" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:53:57.651417 24281 raft_consensus.cc:3037] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.653396 24207 raft_consensus.cc:2454] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 1.
I20240502 01:53:57.653947 24095 leader_election.cc:302] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, 9716594f476e436b91b93ceaed4bcf59; no voters:
I20240502 01:53:57.654071 24281 raft_consensus.cc:2454] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 1.
I20240502 01:53:57.654496 24368 raft_consensus.cc:2781] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:57.654644 24368 raft_consensus.cc:686] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Becoming Leader. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:53:57.654870 24368 consensus_queue.cc:237] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:57.655246 24353 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.658661 24028 catalog_manager.cc:5484] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 227af58cb9554c438ae592baff6af4fb (127.20.65.65). New cstate: current_term: 1 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.709861 24081 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:57.719596 24362 consensus_queue.cc:1035] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.724997 24367 consensus_queue.cc:1035] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.729899 24207 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1edcd38b8d8643ea896111553e49c16c"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
mode: GRACEFUL
new_leader_uuid: "a6f86494e4de4d3c851c5dab07b40568"
from {username='slave'} at 127.0.0.1:32776
I20240502 01:53:57.730091 24207 raft_consensus.cc:595] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Received request to transfer leadership to TS a6f86494e4de4d3c851c5dab07b40568
I20240502 01:53:57.731191 24207 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
mode: GRACEFUL
new_leader_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:32776
I20240502 01:53:57.731401 24207 raft_consensus.cc:595] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Received request to transfer leadership to TS 227af58cb9554c438ae592baff6af4fb
I20240502 01:53:57.731806 24081 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 01:53:57.732026 24081 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 01:53:57.749713 24186 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:57.753122 24184 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:57.754002 24184 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
I20240502 01:53:57.762195 24280 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.762776 24279 raft_consensus.cc:1261] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.762837 24129 raft_consensus.cc:1261] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.763273 24131 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.763331 24128 raft_consensus.cc:1261] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:57.763774 24130 raft_consensus.cc:1261] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.764173 24281 raft_consensus.cc:1261] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.764576 24362 consensus_queue.cc:1035] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.765079 24354 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.765033 24362 consensus_queue.cc:1035] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:53:57.766336 24333 consensus_queue.cc:1035] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.767267 24333 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.765902 24355 consensus_queue.cc:1035] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 01:53:57.768752 24353 consensus_queue.cc:1035] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:57.790822 24184 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
I20240502 01:53:57.792898 24206 raft_consensus.cc:1261] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240502 01:53:57.794263 24177 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:57.794564 24183 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
I20240502 01:53:57.794687 24207 raft_consensus.cc:1261] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:53:57.794982 24181 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
I20240502 01:53:57.795882 24327 consensus_queue.cc:1035] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.796573 24327 consensus_queue.cc:1035] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.835016 24353 consensus_queue.cc:1035] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:57.840288 24183 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
I20240502 01:53:57.855491 24278 raft_consensus.cc:1261] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.857161 24204 raft_consensus.cc:1261] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:53:57.858033 24183 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:57.859233 24183 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:57.861074 24183 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
I20240502 01:53:57.866518 24336 consensus_queue.cc:1035] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.866803 24368 consensus_queue.cc:1035] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.868793 24278 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.868953 24203 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.869748 24368 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.869998 24277 raft_consensus.cc:1261] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.871327 24203 raft_consensus.cc:1261] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.871989 24336 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.874706 24354 consensus_queue.cc:1035] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.875953 24390 consensus_queue.cc:1035] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.876240 24336 consensus_queue.cc:1035] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.879324 24126 raft_consensus.cc:1261] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.896333 24353 consensus_queue.cc:1035] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:57.906914 24186 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:57.916889 24186 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:57.916918 24181 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:57.917498 24183 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:57.945977 24183 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:57.955359 24181 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:57.959779 24186 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:57.988322 24185 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:57.988513 24186 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:57.991744 24181 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
I20240502 01:53:58.014235 24333 raft_consensus.cc:979] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59: : Instructing follower 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:53:58.014465 24333 raft_consensus.cc:1067] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Signalling peer 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:53:58.022686 24124 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:45068
I20240502 01:53:58.022991 24124 raft_consensus.cc:482] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:58.023157 24124 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:58.034605 24124 raft_consensus.cc:504] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:58.036077 24202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:53:58.036324 24202 raft_consensus.cc:3032] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:58.036448 24202 raft_consensus.cc:729] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:53:58.036785 24202 consensus_queue.cc:260] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:53:58.037209 24202 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:58.039311 24124 leader_election.cc:288] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 2 election: Requested vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:53:58.039467 24274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:53:58.039793 24274 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:53:58.087985 24182 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:58.089838 24180 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Illegal state: replica 9716594f476e436b91b93ceaed4bcf59 is not leader of this config: current role FOLLOWER
W20240502 01:53:58.104321 24422 log.cc:924] Time spent T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
W20240502 01:53:58.114742 24182 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:58.129582 24180 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Illegal state: replica 9716594f476e436b91b93ceaed4bcf59 is not leader of this config: current role FOLLOWER
W20240502 01:53:58.136389 24181 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Illegal state: replica 9716594f476e436b91b93ceaed4bcf59 is not leader of this config: current role FOLLOWER
W20240502 01:53:58.142702 24185 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:58.155591 24182 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:58.169492 24180 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Illegal state: replica 9716594f476e436b91b93ceaed4bcf59 is not leader of this config: current role FOLLOWER
I20240502 01:53:58.180299 24450 mvcc.cc:205] Tried to move back new op lower bound from 7023062375854649344 to 7023062374861492224. Current Snapshot: MvccSnapshot[applied={T|T < 7023062375854649344}]
W20240502 01:53:58.209949 24180 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:58.215627 24108 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45070: Illegal state: replica 227af58cb9554c438ae592baff6af4fb is not leader of this config: current role FOLLOWER
W20240502 01:53:58.216197 24106 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45070: Illegal state: replica 227af58cb9554c438ae592baff6af4fb is not leader of this config: current role FOLLOWER
I20240502 01:53:58.217099 24202 raft_consensus.cc:2454] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 2.
W20240502 01:53:58.217809 24182 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:58.230082 24181 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:58.233839 24105 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45070: Illegal state: replica 227af58cb9554c438ae592baff6af4fb is not leader of this config: current role FOLLOWER
I20240502 01:53:58.234153 24274 raft_consensus.cc:2454] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 2.
I20240502 01:53:58.235918 24096 leader_election.cc:302] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, a6f86494e4de4d3c851c5dab07b40568; no voters:
W20240502 01:53:58.237409 24111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45070: Illegal state: replica 227af58cb9554c438ae592baff6af4fb is not leader of this config: current role FOLLOWER
I20240502 01:53:58.241015 24368 raft_consensus.cc:2781] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:58.241164 24368 raft_consensus.cc:686] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Becoming Leader. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
W20240502 01:53:58.241410 24260 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55090: Illegal state: replica a6f86494e4de4d3c851c5dab07b40568 is not leader of this config: current role FOLLOWER
I20240502 01:53:58.241494 24368 consensus_queue.cc:237] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
W20240502 01:53:58.241799 24255 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55090: Illegal state: replica a6f86494e4de4d3c851c5dab07b40568 is not leader of this config: current role FOLLOWER
W20240502 01:53:58.240626 24172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.65.67:33253, user_credentials={real_user=slave}} blocked reactor thread for 60357us
W20240502 01:53:58.243861 24180 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:58.245587 24255 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55090: Illegal state: replica a6f86494e4de4d3c851c5dab07b40568 is not leader of this config: current role FOLLOWER
W20240502 01:53:58.245587 24182 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
I20240502 01:53:58.259171 24472 raft_consensus.cc:979] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59: : Instructing follower a6f86494e4de4d3c851c5dab07b40568 to start an election
I20240502 01:53:58.259320 24355 raft_consensus.cc:1067] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Signalling peer a6f86494e4de4d3c851c5dab07b40568 to start an election
W20240502 01:53:58.268707 24182 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:58.268792 24180 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
I20240502 01:53:58.271447 24277 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1edcd38b8d8643ea896111553e49c16c"
dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
from {username='slave'} at 127.0.0.1:55076
W20240502 01:53:58.276127 24255 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55090: Illegal state: replica a6f86494e4de4d3c851c5dab07b40568 is not leader of this config: current role FOLLOWER
I20240502 01:53:58.276626 24277 raft_consensus.cc:482] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:58.276794 24277 raft_consensus.cc:3037] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:53:58.277403 24180 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
W20240502 01:53:58.277997 24182 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Service unavailable: leader transfer in progress
I20240502 01:53:58.279901 24277 raft_consensus.cc:504] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } }
I20240502 01:53:58.280797 24277 leader_election.cc:288] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 2 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:53:58.280990 24019 catalog_manager.cc:5484] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb reported cstate change: term changed from 1 to 2, leader changed from 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) to 227af58cb9554c438ae592baff6af4fb (127.20.65.65). New cstate: current_term: 2 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:58.281610 24127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edcd38b8d8643ea896111553e49c16c" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:53:58.281816 24127 raft_consensus.cc:3037] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:58.283578 24202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edcd38b8d8643ea896111553e49c16c" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:53:58.283784 24202 raft_consensus.cc:3032] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:58.283893 24202 raft_consensus.cc:729] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:53:58.284145 24202 consensus_queue.cc:260] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } }
I20240502 01:53:58.284499 24202 raft_consensus.cc:3037] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:58.284727 24127 raft_consensus.cc:2454] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 2.
I20240502 01:53:58.285208 24246 leader_election.cc:302] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:53:58.285506 24362 raft_consensus.cc:2781] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:58.285686 24362 raft_consensus.cc:686] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Becoming Leader. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Running, Role: LEADER
I20240502 01:53:58.285936 24362 consensus_queue.cc:237] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } }
W20240502 01:53:58.286099 24181 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Illegal state: replica 9716594f476e436b91b93ceaed4bcf59 is not leader of this config: current role FOLLOWER
W20240502 01:53:58.286875 24185 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Illegal state: replica 9716594f476e436b91b93ceaed4bcf59 is not leader of this config: current role FOLLOWER
I20240502 01:53:58.287304 24202 raft_consensus.cc:2454] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 2.
W20240502 01:53:58.287597 24182 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Illegal state: replica 9716594f476e436b91b93ceaed4bcf59 is not leader of this config: current role FOLLOWER
W20240502 01:53:58.288282 24180 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Illegal state: replica 9716594f476e436b91b93ceaed4bcf59 is not leader of this config: current role FOLLOWER
W20240502 01:53:58.292631 24180 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Illegal state: replica 9716594f476e436b91b93ceaed4bcf59 is not leader of this config: current role FOLLOWER
W20240502 01:53:58.293778 24180 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32782: Illegal state: replica 9716594f476e436b91b93ceaed4bcf59 is not leader of this config: current role FOLLOWER
I20240502 01:53:58.297061 24019 catalog_manager.cc:5484] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: term changed from 1 to 2, leader changed from 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) to a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67). New cstate: current_term: 2 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:58.297448 24127 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:53:58.297461 24202 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:53:58.298487 24277 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:53:58.298586 24362 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:58.299486 24362 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:58.301234 24390 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:58.304731 24202 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:53:58.307122 24390 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:59.076907 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:59.079948 24485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:59.087843 24486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:59.088927 24488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:59.089227 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:59.089458 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:59.089504 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:59.089526 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614839089523 us; error 0 us; skew 500 ppm
I20240502 01:53:59.089627 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:59.090905 20741 webserver.cc:460] Webserver started at http://127.20.65.68:37885/ using document root <none> and password file <none>
I20240502 01:53:59.091097 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:59.091153 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:59.091240 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:59.091807 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-3-root/instance:
uuid: "cabbb33d9a564f7ca7c2d5cceeedc477"
format_stamp: "Formatted at 2024-05-02 01:53:59 on dist-test-slave-7cj2"
I20240502 01:53:59.094130 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:59.095588 24494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:59.095887 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:53:59.095966 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-3-root
uuid: "cabbb33d9a564f7ca7c2d5cceeedc477"
format_stamp: "Formatted at 2024-05-02 01:53:59 on dist-test-slave-7cj2"
I20240502 01:53:59.096055 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:59.101789 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:59.102192 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:59.102675 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:59.103449 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:59.103505 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:59.103572 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:59.103610 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:59.118490 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.68:38415
I20240502 01:53:59.118506 24556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.68:38415 every 8 connection(s)
I20240502 01:53:59.122135 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:59.126475 24562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:59.127594 24561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:59.128305 24557 heartbeater.cc:347] Connected to a master server at 127.20.65.126:36935
I20240502 01:53:59.128468 24557 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:59.128994 24557 heartbeater.cc:511] Master 127.20.65.126:36935 requested a full tablet report, sending...
I20240502 01:53:59.130121 24019 ts_manager.cc:194] Registered new tserver with Master: cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68:38415)
W20240502 01:53:59.130818 24564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:59.131157 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:59.131191 24019 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49258
I20240502 01:53:59.131383 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:59.131459 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:59.131498 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614839131494 us; error 0 us; skew 500 ppm
I20240502 01:53:59.131642 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:59.132997 20741 webserver.cc:460] Webserver started at http://127.20.65.69:41135/ using document root <none> and password file <none>
I20240502 01:53:59.133183 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:59.133251 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:59.133366 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:59.133947 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-4-root/instance:
uuid: "a3bcf2c9cfd24f25964eea216f017565"
format_stamp: "Formatted at 2024-05-02 01:53:59 on dist-test-slave-7cj2"
I20240502 01:53:59.136454 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 01:53:59.137903 24569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:59.138211 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:53:59.138305 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-4-root
uuid: "a3bcf2c9cfd24f25964eea216f017565"
format_stamp: "Formatted at 2024-05-02 01:53:59 on dist-test-slave-7cj2"
I20240502 01:53:59.138424 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:59.159360 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:59.159804 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:59.160310 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:59.161070 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:59.161127 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:59.161199 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:59.161235 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:59.176402 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.69:38267
I20240502 01:53:59.176447 24632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.69:38267 every 8 connection(s)
I20240502 01:53:59.177991 20741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:59.180895 24637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:59.181677 24639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:59.182366 24636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:59.182776 20741 server_base.cc:1034] running on GCE node
I20240502 01:53:59.183073 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:53:59.183173 20741 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:53:59.183238 20741 hybrid_clock.cc:648] HybridClock initialized: now 1714614839183234 us; error 0 us; skew 500 ppm
I20240502 01:53:59.183403 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:59.183923 24633 heartbeater.cc:347] Connected to a master server at 127.20.65.126:36935
I20240502 01:53:59.184043 24633 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:59.184445 24633 heartbeater.cc:511] Master 127.20.65.126:36935 requested a full tablet report, sending...
I20240502 01:53:59.184823 20741 webserver.cc:460] Webserver started at http://127.20.65.70:43011/ using document root <none> and password file <none>
I20240502 01:53:59.185036 20741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:59.185112 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:59.185236 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:59.185307 24019 ts_manager.cc:194] Registered new tserver with Master: a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69:38267)
I20240502 01:53:59.186054 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-5-root/instance:
uuid: "03b70cf3fe3b4a32ad200d00c0c88367"
format_stamp: "Formatted at 2024-05-02 01:53:59 on dist-test-slave-7cj2"
I20240502 01:53:59.186332 24019 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49270
I20240502 01:53:59.188584 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 01:53:59.189939 24644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:59.190223 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:53:59.190315 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-5-root
uuid: "03b70cf3fe3b4a32ad200d00c0c88367"
format_stamp: "Formatted at 2024-05-02 01:53:59 on dist-test-slave-7cj2"
I20240502 01:53:59.190428 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:59.211760 20741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:59.212249 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:59.212731 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:59.213471 20741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:59.213529 20741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:59.213598 20741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:59.213636 20741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:59.228041 20741 rpc_server.cc:307] RPC server started. Bound to: 127.20.65.70:35815
I20240502 01:53:59.228102 24706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.65.70:35815 every 8 connection(s)
I20240502 01:53:59.234680 24707 heartbeater.cc:347] Connected to a master server at 127.20.65.126:36935
I20240502 01:53:59.234803 24707 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:59.235198 24707 heartbeater.cc:511] Master 127.20.65.126:36935 requested a full tablet report, sending...
I20240502 01:53:59.235976 24019 ts_manager.cc:194] Registered new tserver with Master: 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70:35815)
I20240502 01:53:59.236716 24019 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49276
I20240502 01:53:59.732442 24081 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:53:59.744026 24274 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "776becbedf8a4b89a59ea06cf8e2163a"
dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
mode: GRACEFUL
new_leader_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:55106
I20240502 01:53:59.744216 24274 raft_consensus.cc:595] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Received request to transfer leadership to TS 227af58cb9554c438ae592baff6af4fb
I20240502 01:53:59.747893 24207 consensus_queue.cc:237] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } }
I20240502 01:53:59.756362 24129 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:53:59.757010 24129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
mode: GRACEFUL
new_leader_uuid: "9716594f476e436b91b93ceaed4bcf59"
from {username='slave'} at 127.0.0.1:45086
I20240502 01:53:59.757241 24129 raft_consensus.cc:595] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Received request to transfer leadership to TS 9716594f476e436b91b93ceaed4bcf59
I20240502 01:53:59.757401 24481 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:53:59.757792 24081 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 01:53:59.758013 24081 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:53:59.759353 24274 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:53:59.759904 24481 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:53:59.760493 24481 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
W20240502 01:53:59.763418 24171 consensus_peers.cc:485] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 -> Peer 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70:35815): Couldn't send request to peer 03b70cf3fe3b4a32ad200d00c0c88367. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d516fd48dce443169e7e6011ea6a704d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:59.764832 24129 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:53:59.767925 24019 catalog_manager.cc:5484] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: config changed from index -1 to 28, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New cstate: current_term: 1 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:59.769593 24274 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:53:59.771629 24124 consensus_queue.cc:237] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:53:59.787264 24274 raft_consensus.cc:1261] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 01:53:59.787896 24720 consensus_queue.cc:1035] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:53:59.788081 24207 raft_consensus.cc:1261] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 01:53:59.791067 24720 raft_consensus.cc:2932] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
I20240502 01:53:59.794216 24274 raft_consensus.cc:2932] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
I20240502 01:53:59.795344 24124 consensus_queue.cc:237] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:53:59.801674 24022 catalog_manager.cc:5484] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: config changed from index -1 to 26, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New cstate: current_term: 1 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
I20240502 01:53:59.832182 24728 consensus_queue.cc:1035] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:53:59.833838 24207 raft_consensus.cc:2932] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
W20240502 01:53:59.839066 24095 consensus_peers.cc:485] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb -> Peer cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68:38415): Couldn't send request to peer cabbb33d9a564f7ca7c2d5cceeedc477. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3f5d1021b8b4932b7fe13b7fb933a9c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:59.841176 24274 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:53:59.841176 24207 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:53:59.841754 24727 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:53:59.842051 24726 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:53:59.844319 24476 raft_consensus.cc:2932] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
W20240502 01:53:59.844929 24095 consensus_peers.cc:485] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb -> Peer a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69:38267): Couldn't send request to peer a3bcf2c9cfd24f25964eea216f017565. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2cf1c78d5f3f406d9029fe29deb6412a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:59.845041 24207 raft_consensus.cc:2932] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
I20240502 01:53:59.845191 24281 raft_consensus.cc:2932] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
I20240502 01:53:59.849558 24124 consensus_queue.cc:237] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:53:59.849501 24022 catalog_manager.cc:5484] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: config changed from index -1 to 28, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New cstate: current_term: 1 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
I20240502 01:53:59.851177 24207 raft_consensus.cc:1261] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
W20240502 01:53:59.851462 24095 consensus_peers.cc:485] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb -> Peer cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68:38415): Couldn't send request to peer cabbb33d9a564f7ca7c2d5cceeedc477. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 947791f925a047cf82a9b28983f679bf. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:59.851738 24728 consensus_queue.cc:1035] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:53:59.851872 24281 raft_consensus.cc:1261] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 01:53:59.852468 24728 consensus_queue.cc:1035] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:53:59.854923 24720 raft_consensus.cc:2932] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
I20240502 01:53:59.855532 24281 raft_consensus.cc:2932] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
I20240502 01:53:59.856567 24207 raft_consensus.cc:2932] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
I20240502 01:53:59.859723 24019 catalog_manager.cc:5484] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb reported cstate change: config changed from index -1 to 27, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New cstate: current_term: 1 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:59.865569 24281 consensus_queue.cc:237] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } }
I20240502 01:53:59.868168 24124 raft_consensus.cc:1261] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:53:59.868193 24207 raft_consensus.cc:1261] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:53:59.868688 24483 consensus_queue.cc:1035] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:53:59.869019 24731 consensus_queue.cc:1035] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:53:59.871261 24483 raft_consensus.cc:2932] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:53:59.874538 24207 raft_consensus.cc:2932] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:53:59.875032 24124 raft_consensus.cc:2932] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:53:59.875571 24281 consensus_queue.cc:237] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:53:59.875502 24019 catalog_manager.cc:5484] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: config changed from index -1 to 28, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New cstate: current_term: 1 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:59.879470 24246 consensus_peers.cc:485] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 -> Peer 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70:35815): Couldn't send request to peer 03b70cf3fe3b4a32ad200d00c0c88367. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc63402f38f94f50b33661f6d776f813. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:59.879588 24129 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 01:53:59.880334 24731 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:53:59.881481 24207 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 01:53:59.881976 24731 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:53:59.883272 24731 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
W20240502 01:53:59.887604 24246 consensus_peers.cc:485] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 -> Peer a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69:38267): Couldn't send request to peer a3bcf2c9cfd24f25964eea216f017565. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1edcd38b8d8643ea896111553e49c16c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:59.887496 24124 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
I20240502 01:53:59.888131 24203 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
I20240502 01:53:59.891953 24027 catalog_manager.cc:5484] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: config changed from index -1 to 28, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New cstate: current_term: 2 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:00.023432 24727 raft_consensus.cc:979] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb: : Instructing follower 9716594f476e436b91b93ceaed4bcf59 to start an election
I20240502 01:54:00.023598 24727 raft_consensus.cc:1067] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Signalling peer 9716594f476e436b91b93ceaed4bcf59 to start an election
I20240502 01:54:00.024148 24203 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
from {username='slave'} at 127.0.0.1:60992
I20240502 01:54:00.024318 24203 raft_consensus.cc:482] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:00.024422 24203 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:00.026649 24203 raft_consensus.cc:504] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:00.027233 24203 leader_election.cc:288] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 3 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:54:00.027626 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:54:00.027626 24129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:54:00.027845 24129 raft_consensus.cc:3032] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:00.027845 24281 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:00.027961 24129 raft_consensus.cc:729] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:00.028223 24129 consensus_queue.cc:260] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:00.028571 24129 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:00.030683 24281 raft_consensus.cc:2454] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 3.
I20240502 01:54:00.031096 24172 leader_election.cc:302] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9716594f476e436b91b93ceaed4bcf59, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:54:00.031154 24129 raft_consensus.cc:2454] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 3.
I20240502 01:54:00.031348 24715 raft_consensus.cc:2781] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:00.031467 24715 raft_consensus.cc:686] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Becoming Leader. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:54:00.031697 24715 consensus_queue.cc:237] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:00.033929 24022 catalog_manager.cc:5484] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: term changed from 2 to 3, leader changed from 227af58cb9554c438ae592baff6af4fb (127.20.65.65) to 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66). New cstate: current_term: 3 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:00.054592 24734 raft_consensus.cc:979] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568: : Instructing follower 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:54:00.054728 24734 raft_consensus.cc:1067] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Signalling peer 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:54:00.055289 24129 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "776becbedf8a4b89a59ea06cf8e2163a"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:45062
I20240502 01:54:00.055431 24129 raft_consensus.cc:482] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:00.055526 24129 raft_consensus.cc:3037] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:00.057917 24129 raft_consensus.cc:504] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:00.058619 24129 leader_election.cc:288] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 2 election: Requested vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:54:00.058868 24203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776becbedf8a4b89a59ea06cf8e2163a" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:54:00.058955 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776becbedf8a4b89a59ea06cf8e2163a" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:54:00.059026 24203 raft_consensus.cc:3037] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:00.059090 24281 raft_consensus.cc:3032] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:00.059130 24281 raft_consensus.cc:729] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Running, Role: LEADER
I20240502 01:54:00.059307 24281 consensus_queue.cc:260] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:00.059535 24281 raft_consensus.cc:3037] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:00.060712 24203 raft_consensus.cc:2454] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 2.
I20240502 01:54:00.061041 24095 leader_election.cc:302] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, 9716594f476e436b91b93ceaed4bcf59; no voters:
I20240502 01:54:00.061297 24727 raft_consensus.cc:2781] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:00.061345 24281 raft_consensus.cc:2454] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 2.
I20240502 01:54:00.061486 24727 raft_consensus.cc:686] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Becoming Leader. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:00.061759 24727 consensus_queue.cc:237] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:00.064193 24022 catalog_manager.cc:5484] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb reported cstate change: term changed from 1 to 2, leader changed from a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) to 227af58cb9554c438ae592baff6af4fb (127.20.65.65). New cstate: current_term: 2 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:00.132455 24557 heartbeater.cc:503] Master 127.20.65.126:36935 was elected leader, sending a full tablet report...
I20240502 01:54:00.187462 24633 heartbeater.cc:503] Master 127.20.65.126:36935 was elected leader, sending a full tablet report...
I20240502 01:54:00.237774 24707 heartbeater.cc:503] Master 127.20.65.126:36935 was elected leader, sending a full tablet report...
I20240502 01:54:00.252454 24735 ts_tablet_manager.cc:902] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367: Initiating tablet copy from peer 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:54:00.253921 24735 tablet_copy_client.cc:308] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Beginning tablet copy session from remote peer at address 127.20.65.66:37365
I20240502 01:54:00.274331 24217 tablet_copy_service.cc:140] P 9716594f476e436b91b93ceaed4bcf59: Received BeginTabletCopySession request for tablet d516fd48dce443169e7e6011ea6a704d from peer 03b70cf3fe3b4a32ad200d00c0c88367 ({username='slave'} at 127.0.0.1:32800)
I20240502 01:54:00.274549 24217 tablet_copy_service.cc:161] P 9716594f476e436b91b93ceaed4bcf59: Beginning new tablet copy session on tablet d516fd48dce443169e7e6011ea6a704d from peer 03b70cf3fe3b4a32ad200d00c0c88367 at {username='slave'} at 127.0.0.1:32800: session id = 03b70cf3fe3b4a32ad200d00c0c88367-d516fd48dce443169e7e6011ea6a704d
I20240502 01:54:00.276894 24217 tablet_copy_source_session.cc:205] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:00.278012 24735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d516fd48dce443169e7e6011ea6a704d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.282878 24735 tablet_copy_client.cc:783] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:00.283170 24735 tablet_copy_client.cc:647] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:00.284806 24739 tablet_copy_client.cc:1002] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Injecting latency into file download: 2000
I20240502 01:54:00.303567 24742 ts_tablet_manager.cc:902] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367: Initiating tablet copy from peer a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:54:00.305114 24742 tablet_copy_client.cc:308] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Beginning tablet copy session from remote peer at address 127.20.65.67:33253
I20240502 01:54:00.310472 24291 tablet_copy_service.cc:140] P a6f86494e4de4d3c851c5dab07b40568: Received BeginTabletCopySession request for tablet dc63402f38f94f50b33661f6d776f813 from peer 03b70cf3fe3b4a32ad200d00c0c88367 ({username='slave'} at 127.0.0.1:55122)
I20240502 01:54:00.310607 24291 tablet_copy_service.cc:161] P a6f86494e4de4d3c851c5dab07b40568: Beginning new tablet copy session on tablet dc63402f38f94f50b33661f6d776f813 from peer 03b70cf3fe3b4a32ad200d00c0c88367 at {username='slave'} at 127.0.0.1:55122: session id = 03b70cf3fe3b4a32ad200d00c0c88367-dc63402f38f94f50b33661f6d776f813
I20240502 01:54:00.313207 24291 tablet_copy_source_session.cc:205] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:00.314918 24742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc63402f38f94f50b33661f6d776f813. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.319546 24742 tablet_copy_client.cc:783] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:00.319872 24742 tablet_copy_client.cc:647] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:00.321245 24746 tablet_copy_client.cc:1002] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Injecting latency into file download: 2000
I20240502 01:54:00.362681 24747 ts_tablet_manager.cc:902] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477: Initiating tablet copy from peer 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551)
I20240502 01:54:00.364073 24747 tablet_copy_client.cc:308] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Beginning tablet copy session from remote peer at address 127.20.65.65:43551
I20240502 01:54:00.370517 24141 tablet_copy_service.cc:140] P 227af58cb9554c438ae592baff6af4fb: Received BeginTabletCopySession request for tablet d3f5d1021b8b4932b7fe13b7fb933a9c from peer cabbb33d9a564f7ca7c2d5cceeedc477 ({username='slave'} at 127.0.0.1:45110)
I20240502 01:54:00.370709 24141 tablet_copy_service.cc:161] P 227af58cb9554c438ae592baff6af4fb: Beginning new tablet copy session on tablet d3f5d1021b8b4932b7fe13b7fb933a9c from peer cabbb33d9a564f7ca7c2d5cceeedc477 at {username='slave'} at 127.0.0.1:45110: session id = cabbb33d9a564f7ca7c2d5cceeedc477-d3f5d1021b8b4932b7fe13b7fb933a9c
I20240502 01:54:00.373256 24141 tablet_copy_source_session.cc:205] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:00.375193 24747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f5d1021b8b4932b7fe13b7fb933a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.380023 24747 tablet_copy_client.cc:783] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:00.380204 24747 tablet_copy_client.cc:647] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:00.381685 24751 tablet_copy_client.cc:1002] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Injecting latency into file download: 2000
I20240502 01:54:00.381798 24754 ts_tablet_manager.cc:902] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565: Initiating tablet copy from peer 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551)
I20240502 01:54:00.383548 24754 tablet_copy_client.cc:308] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Beginning tablet copy session from remote peer at address 127.20.65.65:43551
I20240502 01:54:00.389546 24141 tablet_copy_service.cc:140] P 227af58cb9554c438ae592baff6af4fb: Received BeginTabletCopySession request for tablet 2cf1c78d5f3f406d9029fe29deb6412a from peer a3bcf2c9cfd24f25964eea216f017565 ({username='slave'} at 127.0.0.1:45120)
I20240502 01:54:00.389742 24141 tablet_copy_service.cc:161] P 227af58cb9554c438ae592baff6af4fb: Beginning new tablet copy session on tablet 2cf1c78d5f3f406d9029fe29deb6412a from peer a3bcf2c9cfd24f25964eea216f017565 at {username='slave'} at 127.0.0.1:45120: session id = a3bcf2c9cfd24f25964eea216f017565-2cf1c78d5f3f406d9029fe29deb6412a
I20240502 01:54:00.391726 24141 tablet_copy_source_session.cc:205] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:00.392791 24754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf1c78d5f3f406d9029fe29deb6412a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.398171 24754 tablet_copy_client.cc:783] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:00.398569 24754 tablet_copy_client.cc:647] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:00.399977 24758 tablet_copy_client.cc:1002] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Injecting latency into file download: 2000
I20240502 01:54:00.427677 24760 ts_tablet_manager.cc:902] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477: Initiating tablet copy from peer 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551)
I20240502 01:54:00.429072 24760 tablet_copy_client.cc:308] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Beginning tablet copy session from remote peer at address 127.20.65.65:43551
I20240502 01:54:00.429719 24141 tablet_copy_service.cc:140] P 227af58cb9554c438ae592baff6af4fb: Received BeginTabletCopySession request for tablet 947791f925a047cf82a9b28983f679bf from peer cabbb33d9a564f7ca7c2d5cceeedc477 ({username='slave'} at 127.0.0.1:45110)
I20240502 01:54:00.429896 24141 tablet_copy_service.cc:161] P 227af58cb9554c438ae592baff6af4fb: Beginning new tablet copy session on tablet 947791f925a047cf82a9b28983f679bf from peer cabbb33d9a564f7ca7c2d5cceeedc477 at {username='slave'} at 127.0.0.1:45110: session id = cabbb33d9a564f7ca7c2d5cceeedc477-947791f925a047cf82a9b28983f679bf
I20240502 01:54:00.432624 24141 tablet_copy_source_session.cc:205] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:00.433794 24760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947791f925a047cf82a9b28983f679bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.439942 24760 tablet_copy_client.cc:783] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:00.440241 24760 tablet_copy_client.cc:647] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:00.441848 24764 tablet_copy_client.cc:1002] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Injecting latency into file download: 2000
I20240502 01:54:00.472357 24765 ts_tablet_manager.cc:902] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565: Initiating tablet copy from peer a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:54:00.473860 24765 tablet_copy_client.cc:308] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Beginning tablet copy session from remote peer at address 127.20.65.67:33253
I20240502 01:54:00.479593 24291 tablet_copy_service.cc:140] P a6f86494e4de4d3c851c5dab07b40568: Received BeginTabletCopySession request for tablet 1edcd38b8d8643ea896111553e49c16c from peer a3bcf2c9cfd24f25964eea216f017565 ({username='slave'} at 127.0.0.1:55132)
I20240502 01:54:00.479843 24291 tablet_copy_service.cc:161] P a6f86494e4de4d3c851c5dab07b40568: Beginning new tablet copy session on tablet 1edcd38b8d8643ea896111553e49c16c from peer a3bcf2c9cfd24f25964eea216f017565 at {username='slave'} at 127.0.0.1:55132: session id = a3bcf2c9cfd24f25964eea216f017565-1edcd38b8d8643ea896111553e49c16c
I20240502 01:54:00.482235 24291 tablet_copy_source_session.cc:205] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:00.483477 24765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edcd38b8d8643ea896111553e49c16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.488368 24765 tablet_copy_client.cc:783] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:00.488595 24765 tablet_copy_client.cc:647] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:00.490027 24769 tablet_copy_client.cc:1002] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Injecting latency into file download: 2000
I20240502 01:54:00.558948 24124 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 01:54:00.559567 24715 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:00.564899 24281 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 01:54:00.565452 24773 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:00.632536 24207 raft_consensus.cc:1261] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 01:54:00.633035 24728 consensus_queue.cc:1035] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:00.641036 24274 raft_consensus.cc:1261] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 01:54:00.642329 24476 consensus_queue.cc:1035] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:01.758425 24081 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:54:01.761359 24124 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "776becbedf8a4b89a59ea06cf8e2163a"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
mode: GRACEFUL
new_leader_uuid: "a6f86494e4de4d3c851c5dab07b40568"
from {username='slave'} at 127.0.0.1:45086
I20240502 01:54:01.761564 24124 raft_consensus.cc:595] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Received request to transfer leadership to TS a6f86494e4de4d3c851c5dab07b40568
I20240502 01:54:01.761962 24081 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 01:54:01.762173 24081 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:54:02.139499 24780 raft_consensus.cc:979] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb: : Instructing follower a6f86494e4de4d3c851c5dab07b40568 to start an election
I20240502 01:54:02.139690 24780 raft_consensus.cc:1067] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Signalling peer a6f86494e4de4d3c851c5dab07b40568 to start an election
I20240502 01:54:02.140304 24274 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "776becbedf8a4b89a59ea06cf8e2163a"
dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
from {username='slave'} at 127.0.0.1:55086
I20240502 01:54:02.140475 24274 raft_consensus.cc:482] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:02.140564 24274 raft_consensus.cc:3037] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:02.142517 24274 raft_consensus.cc:504] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:02.143033 24274 leader_election.cc:288] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 3 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:54:02.143368 24124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776becbedf8a4b89a59ea06cf8e2163a" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:54:02.143535 24124 raft_consensus.cc:3032] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:02.143579 24124 raft_consensus.cc:729] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:02.143774 24124 consensus_queue.cc:260] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:02.144116 24124 raft_consensus.cc:3037] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:02.146065 24124 raft_consensus.cc:2454] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 3.
I20240502 01:54:02.146334 24203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776becbedf8a4b89a59ea06cf8e2163a" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:54:02.146454 24246 leader_election.cc:302] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:54:02.146553 24203 raft_consensus.cc:3037] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:02.146723 24483 raft_consensus.cc:2781] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:02.147197 24483 raft_consensus.cc:686] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 3 LEADER]: Becoming Leader. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Running, Role: LEADER
I20240502 01:54:02.147442 24483 consensus_queue.cc:237] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:02.148772 24203 raft_consensus.cc:2454] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 3.
I20240502 01:54:02.151194 24019 catalog_manager.cc:5484] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: term changed from 2 to 3, leader changed from 227af58cb9554c438ae592baff6af4fb (127.20.65.65) to a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67). New cstate: current_term: 3 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:02.285189 24735 tablet_copy_client.cc:523] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:02.288156 24735 tablet_bootstrap.cc:492] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367: Bootstrap starting.
I20240502 01:54:02.321611 24742 tablet_copy_client.cc:523] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:02.324957 24735 tablet_bootstrap.cc:492] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=109 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:02.325215 24742 tablet_bootstrap.cc:492] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367: Bootstrap starting.
I20240502 01:54:02.325343 24735 tablet_bootstrap.cc:492] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367: Bootstrap complete.
I20240502 01:54:02.325608 24735 ts_tablet_manager.cc:1367] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367: Time spent bootstrapping tablet: real 0.038s user 0.032s sys 0.004s
I20240502 01:54:02.326304 24735 raft_consensus.cc:348] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } }
I20240502 01:54:02.326578 24735 raft_consensus.cc:729] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03b70cf3fe3b4a32ad200d00c0c88367, State: Initialized, Role: LEARNER
I20240502 01:54:02.326843 24735 consensus_queue.cc:260] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } }
I20240502 01:54:02.327744 24735 ts_tablet_manager.cc:1398] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367: Time spent starting tablet: real 0.002s user 0.001s sys 0.004s
I20240502 01:54:02.330207 24217 tablet_copy_service.cc:342] P 9716594f476e436b91b93ceaed4bcf59: Request end of tablet copy session 03b70cf3fe3b4a32ad200d00c0c88367-d516fd48dce443169e7e6011ea6a704d received from {username='slave'} at 127.0.0.1:32800
I20240502 01:54:02.330312 24217 tablet_copy_service.cc:433] P 9716594f476e436b91b93ceaed4bcf59: ending tablet copy session 03b70cf3fe3b4a32ad200d00c0c88367-d516fd48dce443169e7e6011ea6a704d on tablet d516fd48dce443169e7e6011ea6a704d with peer 03b70cf3fe3b4a32ad200d00c0c88367
I20240502 01:54:02.364408 24742 tablet_bootstrap.cc:492] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=126 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:02.364871 24742 tablet_bootstrap.cc:492] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367: Bootstrap complete.
I20240502 01:54:02.365157 24742 ts_tablet_manager.cc:1367] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367: Time spent bootstrapping tablet: real 0.040s user 0.039s sys 0.000s
I20240502 01:54:02.365871 24742 raft_consensus.cc:348] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } }
I20240502 01:54:02.366103 24742 raft_consensus.cc:729] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03b70cf3fe3b4a32ad200d00c0c88367, State: Initialized, Role: LEARNER
I20240502 01:54:02.366302 24742 consensus_queue.cc:260] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } }
I20240502 01:54:02.366888 24742 ts_tablet_manager.cc:1398] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:54:02.367671 24291 tablet_copy_service.cc:342] P a6f86494e4de4d3c851c5dab07b40568: Request end of tablet copy session 03b70cf3fe3b4a32ad200d00c0c88367-dc63402f38f94f50b33661f6d776f813 received from {username='slave'} at 127.0.0.1:55122
I20240502 01:54:02.367800 24291 tablet_copy_service.cc:433] P a6f86494e4de4d3c851c5dab07b40568: ending tablet copy session 03b70cf3fe3b4a32ad200d00c0c88367-dc63402f38f94f50b33661f6d776f813 on tablet dc63402f38f94f50b33661f6d776f813 with peer 03b70cf3fe3b4a32ad200d00c0c88367
I20240502 01:54:02.382007 24747 tablet_copy_client.cc:523] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:02.385130 24747 tablet_bootstrap.cc:492] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477: Bootstrap starting.
I20240502 01:54:02.400363 24754 tablet_copy_client.cc:523] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:02.403652 24754 tablet_bootstrap.cc:492] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565: Bootstrap starting.
I20240502 01:54:02.419929 24747 tablet_bootstrap.cc:492] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=95 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:02.420235 24747 tablet_bootstrap.cc:492] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477: Bootstrap complete.
I20240502 01:54:02.420454 24747 ts_tablet_manager.cc:1367] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477: Time spent bootstrapping tablet: real 0.035s user 0.031s sys 0.005s
I20240502 01:54:02.421152 24747 raft_consensus.cc:348] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:02.421392 24747 raft_consensus.cc:729] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cabbb33d9a564f7ca7c2d5cceeedc477, State: Initialized, Role: LEARNER
I20240502 01:54:02.421669 24747 consensus_queue.cc:260] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:02.428632 24747 ts_tablet_manager.cc:1398] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477: Time spent starting tablet: real 0.008s user 0.006s sys 0.004s
I20240502 01:54:02.429298 24141 tablet_copy_service.cc:342] P 227af58cb9554c438ae592baff6af4fb: Request end of tablet copy session cabbb33d9a564f7ca7c2d5cceeedc477-d3f5d1021b8b4932b7fe13b7fb933a9c received from {username='slave'} at 127.0.0.1:45110
I20240502 01:54:02.429409 24141 tablet_copy_service.cc:433] P 227af58cb9554c438ae592baff6af4fb: ending tablet copy session cabbb33d9a564f7ca7c2d5cceeedc477-d3f5d1021b8b4932b7fe13b7fb933a9c on tablet d3f5d1021b8b4932b7fe13b7fb933a9c with peer cabbb33d9a564f7ca7c2d5cceeedc477
I20240502 01:54:02.442209 24760 tablet_copy_client.cc:523] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:02.445310 24760 tablet_bootstrap.cc:492] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477: Bootstrap starting.
I20240502 01:54:02.447573 24754 tablet_bootstrap.cc:492] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=103 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:02.447911 24754 tablet_bootstrap.cc:492] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565: Bootstrap complete.
I20240502 01:54:02.448143 24754 ts_tablet_manager.cc:1367] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565: Time spent bootstrapping tablet: real 0.045s user 0.034s sys 0.005s
I20240502 01:54:02.448870 24754 raft_consensus.cc:348] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:02.449069 24754 raft_consensus.cc:729] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a3bcf2c9cfd24f25964eea216f017565, State: Initialized, Role: LEARNER
I20240502 01:54:02.449295 24754 consensus_queue.cc:260] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:02.450166 24754 ts_tablet_manager.cc:1398] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240502 01:54:02.450953 24141 tablet_copy_service.cc:342] P 227af58cb9554c438ae592baff6af4fb: Request end of tablet copy session a3bcf2c9cfd24f25964eea216f017565-2cf1c78d5f3f406d9029fe29deb6412a received from {username='slave'} at 127.0.0.1:45120
I20240502 01:54:02.451061 24141 tablet_copy_service.cc:433] P 227af58cb9554c438ae592baff6af4fb: ending tablet copy session a3bcf2c9cfd24f25964eea216f017565-2cf1c78d5f3f406d9029fe29deb6412a on tablet 2cf1c78d5f3f406d9029fe29deb6412a with peer a3bcf2c9cfd24f25964eea216f017565
I20240502 01:54:02.480311 24760 tablet_bootstrap.cc:492] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=98 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:02.480710 24760 tablet_bootstrap.cc:492] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477: Bootstrap complete.
I20240502 01:54:02.480994 24760 ts_tablet_manager.cc:1367] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477: Time spent bootstrapping tablet: real 0.036s user 0.027s sys 0.007s
I20240502 01:54:02.481665 24760 raft_consensus.cc:348] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:02.481866 24760 raft_consensus.cc:729] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cabbb33d9a564f7ca7c2d5cceeedc477, State: Initialized, Role: LEARNER
I20240502 01:54:02.482080 24760 consensus_queue.cc:260] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:02.482605 24760 ts_tablet_manager.cc:1398] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240502 01:54:02.483217 24141 tablet_copy_service.cc:342] P 227af58cb9554c438ae592baff6af4fb: Request end of tablet copy session cabbb33d9a564f7ca7c2d5cceeedc477-947791f925a047cf82a9b28983f679bf received from {username='slave'} at 127.0.0.1:45110
I20240502 01:54:02.483335 24141 tablet_copy_service.cc:433] P 227af58cb9554c438ae592baff6af4fb: ending tablet copy session cabbb33d9a564f7ca7c2d5cceeedc477-947791f925a047cf82a9b28983f679bf on tablet 947791f925a047cf82a9b28983f679bf with peer cabbb33d9a564f7ca7c2d5cceeedc477
I20240502 01:54:02.490350 24765 tablet_copy_client.cc:523] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:02.493654 24765 tablet_bootstrap.cc:492] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565: Bootstrap starting.
I20240502 01:54:02.496099 24532 raft_consensus.cc:1203] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240502 01:54:02.531016 24765 tablet_bootstrap.cc:492] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=97 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:02.531492 24765 tablet_bootstrap.cc:492] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565: Bootstrap complete.
I20240502 01:54:02.531781 24765 ts_tablet_manager.cc:1367] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565: Time spent bootstrapping tablet: real 0.038s user 0.036s sys 0.004s
I20240502 01:54:02.532459 24765 raft_consensus.cc:348] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:02.532660 24765 raft_consensus.cc:729] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a3bcf2c9cfd24f25964eea216f017565, State: Initialized, Role: LEARNER
I20240502 01:54:02.532886 24765 consensus_queue.cc:260] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:02.533447 24765 ts_tablet_manager.cc:1398] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:02.534108 24291 tablet_copy_service.cc:342] P a6f86494e4de4d3c851c5dab07b40568: Request end of tablet copy session a3bcf2c9cfd24f25964eea216f017565-1edcd38b8d8643ea896111553e49c16c received from {username='slave'} at 127.0.0.1:55132
I20240502 01:54:02.534214 24291 tablet_copy_service.cc:433] P a6f86494e4de4d3c851c5dab07b40568: ending tablet copy session a3bcf2c9cfd24f25964eea216f017565-1edcd38b8d8643ea896111553e49c16c on tablet 1edcd38b8d8643ea896111553e49c16c with peer a3bcf2c9cfd24f25964eea216f017565
I20240502 01:54:02.537065 24532 raft_consensus.cc:1203] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240502 01:54:02.543253 24203 raft_consensus.cc:1261] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 01:54:02.543736 24483 consensus_queue.cc:1035] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:02.547504 24128 raft_consensus.cc:1261] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 01:54:02.548018 24799 consensus_queue.cc:1035] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:02.572854 24608 raft_consensus.cc:1203] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20240502 01:54:02.573330 24607 raft_consensus.cc:1203] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240502 01:54:02.597685 24682 raft_consensus.cc:1203] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240502 01:54:02.666172 24682 raft_consensus.cc:1203] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240502 01:54:02.915236 24780 raft_consensus.cc:1050] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb: attempting to promote NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 to VOTER
I20240502 01:54:02.916075 24780 consensus_queue.cc:237] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:02.918659 24532 raft_consensus.cc:1261] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 LEARNER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:54:02.919098 24281 raft_consensus.cc:1261] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:54:02.919489 24802 consensus_queue.cc:1035] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:02.919787 24780 consensus_queue.cc:1035] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:02.920571 24203 raft_consensus.cc:1261] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:54:02.921172 24802 consensus_queue.cc:1035] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:02.924160 24780 raft_consensus.cc:2932] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:02.924729 24281 raft_consensus.cc:2932] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:02.927194 24203 raft_consensus.cc:2932] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:02.929620 24022 catalog_manager.cc:5484] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb reported cstate change: config changed from index 27 to 28, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:02.931705 24532 raft_consensus.cc:2932] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:02.938937 24124 consensus_queue.cc:237] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:02.940712 24203 raft_consensus.cc:1261] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 01:54:02.940712 24532 raft_consensus.cc:1261] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 01:54:02.941220 24802 consensus_queue.cc:1035] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:02.941538 24805 consensus_queue.cc:1035] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:02.943706 24803 raft_consensus.cc:2932] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:02.944451 24532 raft_consensus.cc:2932] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:02.947382 24014 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 947791f925a047cf82a9b28983f679bf with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:02.948733 24027 catalog_manager.cc:5484] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb reported cstate change: config changed from index 28 to 29, VOTER a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) evicted. New cstate: current_term: 1 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:02.948829 24203 raft_consensus.cc:2932] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:02.953063 24270 tablet_service.cc:1514] Processing DeleteTablet for tablet 947791f925a047cf82a9b28983f679bf with delete_type TABLET_DATA_TOMBSTONED (TS a6f86494e4de4d3c851c5dab07b40568 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:55074
I20240502 01:54:02.953656 24812 tablet_replica.cc:330] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568: stopping tablet replica
I20240502 01:54:02.953920 24812 raft_consensus.cc:2229] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.954345 24812 raft_consensus.cc:2258] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.955224 24812 ts_tablet_manager.cc:1875] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:02.960342 24812 ts_tablet_manager.cc:1888] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240502 01:54:02.960438 24812 log.cc:1196] T 947791f925a047cf82a9b28983f679bf P a6f86494e4de4d3c851c5dab07b40568: Deleting WAL directory at /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-2-root/wals/947791f925a047cf82a9b28983f679bf
I20240502 01:54:02.961119 24015 catalog_manager.cc:4830] TS a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253): tablet 947791f925a047cf82a9b28983f679bf (table test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]) successfully deleted
I20240502 01:54:02.971779 24780 raft_consensus.cc:1050] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb: attempting to promote NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 to VOTER
I20240502 01:54:02.972380 24780 consensus_queue.cc:237] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:02.974361 24532 raft_consensus.cc:1261] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 LEARNER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 01:54:02.974462 24203 raft_consensus.cc:1261] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 01:54:02.974948 24281 raft_consensus.cc:1261] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 01:54:02.974879 24802 consensus_queue.cc:1035] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:54:02.975342 24780 consensus_queue.cc:1035] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:54:02.975601 24803 consensus_queue.cc:1035] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:54:02.980365 24805 raft_consensus.cc:2932] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:02.981050 24274 raft_consensus.cc:2932] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:02.984457 24532 raft_consensus.cc:2932] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:02.987022 24027 catalog_manager.cc:5484] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb reported cstate change: config changed from index 26 to 27, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:02.990075 24203 raft_consensus.cc:2932] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:02.992244 24124 consensus_queue.cc:237] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:02.994661 24532 raft_consensus.cc:1261] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:54:02.995138 24281 raft_consensus.cc:1261] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:54:02.995163 24803 consensus_queue.cc:1035] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:02.995689 24805 consensus_queue.cc:1035] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:02.997380 24803 raft_consensus.cc:2932] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:02.998023 24532 raft_consensus.cc:2932] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:02.999233 24274 raft_consensus.cc:2932] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:03.001811 24014 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d3f5d1021b8b4932b7fe13b7fb933a9c with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:03.003204 24022 catalog_manager.cc:5484] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477 reported cstate change: config changed from index 27 to 28, VOTER 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) evicted. New cstate: current_term: 1 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:03.005133 24483 raft_consensus.cc:1050] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568: attempting to promote NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 to VOTER
I20240502 01:54:03.005849 24483 consensus_queue.cc:237] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } }
I20240502 01:54:03.008360 24194 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f5d1021b8b4932b7fe13b7fb933a9c with delete_type TABLET_DATA_TOMBSTONED (TS 9716594f476e436b91b93ceaed4bcf59 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:60978
I20240502 01:54:03.020936 24818 tablet_replica.cc:330] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59: stopping tablet replica
I20240502 01:54:03.021245 24818 raft_consensus.cc:2229] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.021512 24818 raft_consensus.cc:2258] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.022322 24818 ts_tablet_manager.cc:1875] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:03.022780 24124 raft_consensus.cc:1261] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 01:54:03.022817 24203 raft_consensus.cc:1261] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 01:54:03.023277 24781 consensus_queue.cc:1035] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:03.023468 24682 raft_consensus.cc:1261] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 LEARNER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 01:54:03.023597 24483 consensus_queue.cc:1035] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:03.024252 24781 consensus_queue.cc:1035] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:03.031261 24818 ts_tablet_manager.cc:1888] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240502 01:54:03.031463 24818 log.cc:1196] T d3f5d1021b8b4932b7fe13b7fb933a9c P 9716594f476e436b91b93ceaed4bcf59: Deleting WAL directory at /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-1-root/wals/d3f5d1021b8b4932b7fe13b7fb933a9c
I20240502 01:54:03.032235 24014 catalog_manager.cc:4830] TS 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365): tablet d3f5d1021b8b4932b7fe13b7fb933a9c (table test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]) successfully deleted
I20240502 01:54:03.036581 24799 raft_consensus.cc:2932] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:03.037433 24682 raft_consensus.cc:2932] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:03.038031 24124 raft_consensus.cc:2932] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:03.038789 24819 raft_consensus.cc:1050] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568: attempting to promote NON_VOTER a3bcf2c9cfd24f25964eea216f017565 to VOTER
I20240502 01:54:03.039958 24819 consensus_queue.cc:237] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:03.041357 24203 raft_consensus.cc:2932] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:03.042752 24027 catalog_manager.cc:5484] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: config changed from index 28 to 29, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:03.048441 24203 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 01:54:03.048997 24819 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:03.049160 24607 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [term 2 LEARNER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 01:54:03.048436 24274 consensus_queue.cc:237] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } }
I20240502 01:54:03.049782 24799 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:03.051033 24682 raft_consensus.cc:1261] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 01:54:03.051486 24819 consensus_queue.cc:1035] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:03.052340 24124 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 01:54:03.053330 24781 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:03.055263 24128 raft_consensus.cc:1261] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 01:54:03.055764 24781 consensus_queue.cc:1035] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:03.058192 24781 raft_consensus.cc:2932] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:03.059243 24124 raft_consensus.cc:2932] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:03.060097 24483 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:03.061396 24608 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:03.065310 24015 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dc63402f38f94f50b33661f6d776f813 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:03.066748 24027 catalog_manager.cc:5484] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 reported cstate change: config changed from index 28 to 29, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:03.067572 24022 catalog_manager.cc:5484] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb reported cstate change: config changed from index 29 to 30, VOTER 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) evicted. New cstate: current_term: 1 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:03.058895 24682 raft_consensus.cc:2932] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:03.070693 24207 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:03.072906 24129 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:03.076064 24194 tablet_service.cc:1514] Processing DeleteTablet for tablet dc63402f38f94f50b33661f6d776f813 with delete_type TABLET_DATA_TOMBSTONED (TS 9716594f476e436b91b93ceaed4bcf59 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:60978
I20240502 01:54:03.076781 24818 tablet_replica.cc:330] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59: stopping tablet replica
I20240502 01:54:03.076969 24818 raft_consensus.cc:2229] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.077186 24818 raft_consensus.cc:2258] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.078155 24818 ts_tablet_manager.cc:1875] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:03.079670 24281 consensus_queue.cc:237] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:03.081300 24607 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [term 2 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 01:54:03.081853 24781 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:03.082479 24207 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 01:54:03.083294 24781 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:03.083789 24799 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 227af58cb9554c438ae592baff6af4fb (127.20.65.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:03.084401 24607 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 227af58cb9554c438ae592baff6af4fb (127.20.65.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:03.086406 24207 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 227af58cb9554c438ae592baff6af4fb (127.20.65.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:03.087740 24015 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1edcd38b8d8643ea896111553e49c16c with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:03.089051 24027 catalog_manager.cc:5484] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 reported cstate change: config changed from index 29 to 30, VOTER 227af58cb9554c438ae592baff6af4fb (127.20.65.65) evicted. New cstate: current_term: 2 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:03.091135 24818 ts_tablet_manager.cc:1888] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240502 01:54:03.091286 24818 log.cc:1196] T dc63402f38f94f50b33661f6d776f813 P 9716594f476e436b91b93ceaed4bcf59: Deleting WAL directory at /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-1-root/wals/dc63402f38f94f50b33661f6d776f813
I20240502 01:54:03.091985 24014 catalog_manager.cc:4830] TS 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365): tablet dc63402f38f94f50b33661f6d776f813 (table test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]) successfully deleted
I20240502 01:54:03.092788 24120 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edcd38b8d8643ea896111553e49c16c with delete_type TABLET_DATA_TOMBSTONED (TS 227af58cb9554c438ae592baff6af4fb not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:45056
I20240502 01:54:03.093757 24829 tablet_replica.cc:330] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb: stopping tablet replica
I20240502 01:54:03.093981 24829 raft_consensus.cc:2229] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.094509 24829 raft_consensus.cc:2258] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.095693 24829 ts_tablet_manager.cc:1875] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:03.100991 24829 ts_tablet_manager.cc:1888] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240502 01:54:03.101091 24829 log.cc:1196] T 1edcd38b8d8643ea896111553e49c16c P 227af58cb9554c438ae592baff6af4fb: Deleting WAL directory at /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-0-root/wals/1edcd38b8d8643ea896111553e49c16c
I20240502 01:54:03.101696 24014 catalog_manager.cc:4830] TS 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551): tablet 1edcd38b8d8643ea896111553e49c16c (table test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]) successfully deleted
I20240502 01:54:03.138878 24830 raft_consensus.cc:1050] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59: attempting to promote NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 to VOTER
I20240502 01:54:03.140062 24830 consensus_queue.cc:237] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } }
I20240502 01:54:03.142690 24682 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 LEARNER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 01:54:03.143177 24281 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 01:54:03.143267 24832 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:03.143666 24773 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:03.143767 24129 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 01:54:03.144349 24832 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:03.148248 24773 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:03.148658 24129 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:03.148990 24274 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:03.152112 24682 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:03.154351 24026 catalog_manager.cc:5484] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb reported cstate change: config changed from index 28 to 29, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:03.160066 24207 consensus_queue.cc:237] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } }
I20240502 01:54:03.161497 24682 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 01:54:03.161995 24830 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:03.163481 24274 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 01:54:03.164628 24830 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:03.164409 24832 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 227af58cb9554c438ae592baff6af4fb (127.20.65.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:03.165153 24682 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 227af58cb9554c438ae592baff6af4fb (127.20.65.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:03.167716 24274 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 227af58cb9554c438ae592baff6af4fb (127.20.65.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:03.168898 24014 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d516fd48dce443169e7e6011ea6a704d with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:03.169787 24027 catalog_manager.cc:5484] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: config changed from index 29 to 30, VOTER 227af58cb9554c438ae592baff6af4fb (127.20.65.65) evicted. New cstate: current_term: 1 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:03.173833 24120 tablet_service.cc:1514] Processing DeleteTablet for tablet d516fd48dce443169e7e6011ea6a704d with delete_type TABLET_DATA_TOMBSTONED (TS 227af58cb9554c438ae592baff6af4fb not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:45056
I20240502 01:54:03.174103 24829 tablet_replica.cc:330] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb: stopping tablet replica
I20240502 01:54:03.174273 24829 raft_consensus.cc:2229] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.174659 24829 raft_consensus.cc:2258] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.175551 24829 ts_tablet_manager.cc:1875] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:03.180517 24829 ts_tablet_manager.cc:1888] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240502 01:54:03.180634 24829 log.cc:1196] T d516fd48dce443169e7e6011ea6a704d P 227af58cb9554c438ae592baff6af4fb: Deleting WAL directory at /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-0-root/wals/d516fd48dce443169e7e6011ea6a704d
I20240502 01:54:03.181362 24014 catalog_manager.cc:4830] TS 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551): tablet d516fd48dce443169e7e6011ea6a704d (table test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]) successfully deleted
I20240502 01:54:03.194975 24802 raft_consensus.cc:1050] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb: attempting to promote NON_VOTER a3bcf2c9cfd24f25964eea216f017565 to VOTER
I20240502 01:54:03.195791 24802 consensus_queue.cc:237] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:03.198156 24274 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 01:54:03.198161 24607 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 1 LEARNER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 01:54:03.198642 24207 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 01:54:03.198727 24802 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:03.199002 24805 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:03.199285 24803 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:03.203330 24805 raft_consensus.cc:2932] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:03.206212 24207 raft_consensus.cc:2932] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:03.207717 24607 raft_consensus.cc:2932] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:03.208139 24022 catalog_manager.cc:5484] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb reported cstate change: config changed from index 28 to 29, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:03.215510 24129 consensus_queue.cc:237] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:03.217739 24207 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 01:54:03.217808 24607 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 1 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240502 01:54:03.218199 24802 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:03.218513 24803 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:03.221032 24803 raft_consensus.cc:2932] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:03.222939 24203 raft_consensus.cc:2932] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:03.223830 24014 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2cf1c78d5f3f406d9029fe29deb6412a with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:03.225138 24019 catalog_manager.cc:5484] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb reported cstate change: config changed from index 29 to 30, VOTER a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) evicted. New cstate: current_term: 1 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:03.226469 24607 raft_consensus.cc:2932] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:03.228477 24270 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cf1c78d5f3f406d9029fe29deb6412a with delete_type TABLET_DATA_TOMBSTONED (TS a6f86494e4de4d3c851c5dab07b40568 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:55074
I20240502 01:54:03.228787 24812 tablet_replica.cc:330] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568: stopping tablet replica
I20240502 01:54:03.228951 24812 raft_consensus.cc:2229] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.229084 24812 pending_rounds.cc:62] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568: Trying to abort 1 pending ops.
I20240502 01:54:03.229213 24812 pending_rounds.cc:68] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568: Aborting op as it isn't in flight: id { term: 1 index: 29 } timestamp: 7023062397728002048 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a" old_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } } new_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } } }
I20240502 01:54:03.229597 24812 raft_consensus.cc:2894] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Aborting config change with OpId 1.29: Aborted: Op aborted
I20240502 01:54:03.229729 24812 raft_consensus.cc:2866] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:54:03.229830 24812 raft_consensus.cc:2258] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.230667 24812 ts_tablet_manager.cc:1875] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:03.234994 24812 ts_tablet_manager.cc:1888] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240502 01:54:03.235106 24812 log.cc:1196] T 2cf1c78d5f3f406d9029fe29deb6412a P a6f86494e4de4d3c851c5dab07b40568: Deleting WAL directory at /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-2-root/wals/2cf1c78d5f3f406d9029fe29deb6412a
I20240502 01:54:03.235800 24015 catalog_manager.cc:4830] TS a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253): tablet 2cf1c78d5f3f406d9029fe29deb6412a (table test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]) successfully deleted
I20240502 01:54:03.762550 24081 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:54:03.765265 24274 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1edcd38b8d8643ea896111553e49c16c"
dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
mode: GRACEFUL
new_leader_uuid: "a3bcf2c9cfd24f25964eea216f017565"
from {username='slave'} at 127.0.0.1:55106
I20240502 01:54:03.765439 24274 raft_consensus.cc:595] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Received request to transfer leadership to TS a3bcf2c9cfd24f25964eea216f017565
I20240502 01:54:03.766371 24203 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "68d83f47ca7c4a5caeb4ffda5ebc9757"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
mode: GRACEFUL
new_leader_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:32776
I20240502 01:54:03.766546 24203 raft_consensus.cc:595] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Received request to transfer leadership to TS 227af58cb9554c438ae592baff6af4fb
I20240502 01:54:03.767393 24203 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73d4415a5e104edcbfc621ab5604c431"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
mode: GRACEFUL
new_leader_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:32776
I20240502 01:54:03.767525 24203 raft_consensus.cc:595] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Received request to transfer leadership to TS 227af58cb9554c438ae592baff6af4fb
I20240502 01:54:03.768389 24129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "947791f925a047cf82a9b28983f679bf"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
mode: GRACEFUL
new_leader_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477"
from {username='slave'} at 127.0.0.1:45086
I20240502 01:54:03.768538 24129 raft_consensus.cc:595] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Received request to transfer leadership to TS cabbb33d9a564f7ca7c2d5cceeedc477
I20240502 01:54:03.769348 24203 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d516fd48dce443169e7e6011ea6a704d"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
mode: GRACEFUL
new_leader_uuid: "03b70cf3fe3b4a32ad200d00c0c88367"
from {username='slave'} at 127.0.0.1:32776
I20240502 01:54:03.769470 24203 raft_consensus.cc:595] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Received request to transfer leadership to TS 03b70cf3fe3b4a32ad200d00c0c88367
I20240502 01:54:03.770244 24274 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc63402f38f94f50b33661f6d776f813"
dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
mode: GRACEFUL
new_leader_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:55106
I20240502 01:54:03.770370 24274 raft_consensus.cc:595] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Received request to transfer leadership to TS 227af58cb9554c438ae592baff6af4fb
I20240502 01:54:03.770694 24081 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240502 01:54:03.770907 24081 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:54:03.995190 24840 raft_consensus.cc:979] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59: : Instructing follower 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:54:03.995347 24840 raft_consensus.cc:1067] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Signalling peer 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:54:03.995894 24129 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68d83f47ca7c4a5caeb4ffda5ebc9757"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:45068
I20240502 01:54:03.996057 24129 raft_consensus.cc:482] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:03.996140 24129 raft_consensus.cc:3037] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:03.998034 24129 raft_consensus.cc:504] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:03.998531 24129 leader_election.cc:288] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 2 election: Requested vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:54:03.998992 24274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d83f47ca7c4a5caeb4ffda5ebc9757" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:54:03.999063 24203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d83f47ca7c4a5caeb4ffda5ebc9757" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:54:03.999137 24274 raft_consensus.cc:3037] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:03.999256 24203 raft_consensus.cc:3032] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:03.999325 24203 raft_consensus.cc:729] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:54:03.999527 24203 consensus_queue.cc:260] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:03.999825 24203 raft_consensus.cc:3037] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:04.001713 24203 raft_consensus.cc:2454] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 2.
I20240502 01:54:04.002132 24095 leader_election.cc:302] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, 9716594f476e436b91b93ceaed4bcf59; no voters:
I20240502 01:54:04.002353 24803 raft_consensus.cc:2781] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:04.002720 24803 raft_consensus.cc:686] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Becoming Leader. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:04.003051 24803 consensus_queue.cc:237] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:04.005568 24022 catalog_manager.cc:5484] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb reported cstate change: term changed from 1 to 2, leader changed from 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) to 227af58cb9554c438ae592baff6af4fb (127.20.65.65). New cstate: current_term: 2 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:04.011252 24274 raft_consensus.cc:2454] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 2.
I20240502 01:54:04.039264 24781 raft_consensus.cc:979] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568: : Instructing follower 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:54:04.039389 24781 raft_consensus.cc:1067] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Signalling peer 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:54:04.039964 24129 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc63402f38f94f50b33661f6d776f813"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:45062
I20240502 01:54:04.040129 24129 raft_consensus.cc:482] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:04.040217 24129 raft_consensus.cc:3037] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:04.041975 24129 raft_consensus.cc:504] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } }
I20240502 01:54:04.042615 24129 leader_election.cc:288] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 2 election: Requested vote from peers a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253), 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70:35815)
I20240502 01:54:04.043031 24274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc63402f38f94f50b33661f6d776f813" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:54:04.043323 24274 raft_consensus.cc:3032] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:04.043449 24274 raft_consensus.cc:729] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Running, Role: LEADER
I20240502 01:54:04.043758 24274 consensus_queue.cc:260] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } }
I20240502 01:54:04.044041 24274 raft_consensus.cc:3037] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:04.045727 24274 raft_consensus.cc:2454] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 2.
I20240502 01:54:04.046061 24096 leader_election.cc:302] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:54:04.046329 24803 raft_consensus.cc:2781] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:04.046473 24803 raft_consensus.cc:686] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Becoming Leader. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:04.046703 24803 consensus_queue.cc:237] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } }
I20240502 01:54:04.048435 24682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc63402f38f94f50b33661f6d776f813" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "03b70cf3fe3b4a32ad200d00c0c88367"
I20240502 01:54:04.048563 24682 raft_consensus.cc:3037] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:04.050206 24682 raft_consensus.cc:2454] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 2.
I20240502 01:54:04.051200 24022 catalog_manager.cc:5484] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb reported cstate change: term changed from 1 to 2, leader changed from a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) to 227af58cb9554c438ae592baff6af4fb (127.20.65.65). New cstate: current_term: 2 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:04.057404 24842 raft_consensus.cc:979] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb: : Instructing follower cabbb33d9a564f7ca7c2d5cceeedc477 to start an election
I20240502 01:54:04.057540 24842 raft_consensus.cc:1067] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Signalling peer cabbb33d9a564f7ca7c2d5cceeedc477 to start an election
I20240502 01:54:04.058045 24532 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "947791f925a047cf82a9b28983f679bf"
dest_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477"
from {username='slave'} at 127.0.0.1:55860
I20240502 01:54:04.058171 24532 raft_consensus.cc:482] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:04.058251 24532 raft_consensus.cc:3037] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:04.059877 24532 raft_consensus.cc:504] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:04.060441 24532 leader_election.cc:288] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [CANDIDATE]: Term 2 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:54:04.060773 24129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947791f925a047cf82a9b28983f679bf" candidate_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:54:04.060937 24129 raft_consensus.cc:3032] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:04.061000 24129 raft_consensus.cc:729] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:04.061252 24129 consensus_queue.cc:260] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:04.061558 24129 raft_consensus.cc:3037] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:04.063573 24129 raft_consensus.cc:2454] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cabbb33d9a564f7ca7c2d5cceeedc477 in term 2.
I20240502 01:54:04.063932 24497 leader_election.cc:302] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, cabbb33d9a564f7ca7c2d5cceeedc477; no voters:
I20240502 01:54:04.064476 24852 raft_consensus.cc:2781] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:04.064828 24852 raft_consensus.cc:686] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 LEADER]: Becoming Leader. State: Replica: cabbb33d9a564f7ca7c2d5cceeedc477, State: Running, Role: LEADER
I20240502 01:54:04.065115 24852 consensus_queue.cc:237] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:04.066694 24203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947791f925a047cf82a9b28983f679bf" candidate_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:54:04.066840 24203 raft_consensus.cc:3037] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:04.067586 24022 catalog_manager.cc:5484] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 reported cstate change: term changed from 1 to 2, leader changed from 227af58cb9554c438ae592baff6af4fb (127.20.65.65) to cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68). New cstate: current_term: 2 leader_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:04.068846 24203 raft_consensus.cc:2454] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cabbb33d9a564f7ca7c2d5cceeedc477 in term 2.
I20240502 01:54:04.196239 24841 raft_consensus.cc:979] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59: : Instructing follower 03b70cf3fe3b4a32ad200d00c0c88367 to start an election
I20240502 01:54:04.196426 24840 raft_consensus.cc:1067] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Signalling peer 03b70cf3fe3b4a32ad200d00c0c88367 to start an election
I20240502 01:54:04.197072 24682 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d516fd48dce443169e7e6011ea6a704d"
dest_uuid: "03b70cf3fe3b4a32ad200d00c0c88367"
from {username='slave'} at 127.0.0.1:57572
I20240502 01:54:04.197253 24682 raft_consensus.cc:482] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:04.197351 24682 raft_consensus.cc:3037] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:04.199265 24682 raft_consensus.cc:504] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } }
I20240502 01:54:04.199990 24682 leader_election.cc:288] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [CANDIDATE]: Term 2 election: Requested vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:54:04.200426 24274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d516fd48dce443169e7e6011ea6a704d" candidate_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:54:04.200435 24203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d516fd48dce443169e7e6011ea6a704d" candidate_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:54:04.200577 24274 raft_consensus.cc:3037] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:04.200707 24203 raft_consensus.cc:3032] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:04.200783 24203 raft_consensus.cc:729] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:54:04.200997 24203 consensus_queue.cc:260] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } }
I20240502 01:54:04.201272 24203 raft_consensus.cc:3037] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:04.202535 24274 raft_consensus.cc:2454] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03b70cf3fe3b4a32ad200d00c0c88367 in term 2.
I20240502 01:54:04.202860 24648 leader_election.cc:302] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03b70cf3fe3b4a32ad200d00c0c88367, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:54:04.203212 24203 raft_consensus.cc:2454] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03b70cf3fe3b4a32ad200d00c0c88367 in term 2.
I20240502 01:54:04.203383 24855 raft_consensus.cc:2781] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:04.203742 24855 raft_consensus.cc:686] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [term 2 LEADER]: Becoming Leader. State: Replica: 03b70cf3fe3b4a32ad200d00c0c88367, State: Running, Role: LEADER
I20240502 01:54:04.203974 24855 consensus_queue.cc:237] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } }
I20240502 01:54:04.206197 24019 catalog_manager.cc:5484] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 reported cstate change: term changed from 1 to 2, leader changed from 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) to 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70). New cstate: current_term: 2 leader_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:04.209853 24841 raft_consensus.cc:979] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59: : Instructing follower 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:54:04.209959 24841 raft_consensus.cc:1067] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Signalling peer 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:54:04.210507 24129 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73d4415a5e104edcbfc621ab5604c431"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:45068
I20240502 01:54:04.210655 24129 raft_consensus.cc:482] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:04.210753 24129 raft_consensus.cc:3037] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:04.212375 24129 raft_consensus.cc:504] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:04.212940 24129 leader_election.cc:288] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 2 election: Requested vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:54:04.213344 24203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d4415a5e104edcbfc621ab5604c431" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:54:04.213508 24274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d4415a5e104edcbfc621ab5604c431" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:54:04.213508 24203 raft_consensus.cc:3032] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:04.213686 24203 raft_consensus.cc:729] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:54:04.213707 24274 raft_consensus.cc:3037] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:04.213897 24203 consensus_queue.cc:260] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:04.214166 24203 raft_consensus.cc:3037] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:04.215528 24274 raft_consensus.cc:2454] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 2.
I20240502 01:54:04.215845 24096 leader_election.cc:302] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:54:04.215901 24203 raft_consensus.cc:2454] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 2.
I20240502 01:54:04.216046 24842 raft_consensus.cc:2781] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:04.216156 24842 raft_consensus.cc:686] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Becoming Leader. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:04.216346 24842 consensus_queue.cc:237] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:04.218477 24019 catalog_manager.cc:5484] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb reported cstate change: term changed from 1 to 2, leader changed from 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) to 227af58cb9554c438ae592baff6af4fb (127.20.65.65). New cstate: current_term: 2 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:04.225661 24847 raft_consensus.cc:979] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568: : Instructing follower a3bcf2c9cfd24f25964eea216f017565 to start an election
I20240502 01:54:04.225801 24847 raft_consensus.cc:1067] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Signalling peer a3bcf2c9cfd24f25964eea216f017565 to start an election
I20240502 01:54:04.226486 24607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1edcd38b8d8643ea896111553e49c16c"
dest_uuid: "a3bcf2c9cfd24f25964eea216f017565"
from {username='slave'} at 127.0.0.1:54110
I20240502 01:54:04.226660 24607 raft_consensus.cc:482] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:04.226752 24607 raft_consensus.cc:3037] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:04.228946 24607 raft_consensus.cc:504] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:04.229566 24607 leader_election.cc:288] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [CANDIDATE]: Term 3 election: Requested vote from peers a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253), 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:54:04.230248 24274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edcd38b8d8643ea896111553e49c16c" candidate_uuid: "a3bcf2c9cfd24f25964eea216f017565" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:54:04.230511 24274 raft_consensus.cc:3032] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:04.230602 24274 raft_consensus.cc:729] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Running, Role: LEADER
I20240502 01:54:04.230813 24274 consensus_queue.cc:260] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:04.231089 24274 raft_consensus.cc:3037] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:04.231551 24281 consensus_queue.cc:237] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:04.233935 24129 raft_consensus.cc:1261] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
W20240502 01:54:04.234122 24246 consensus_peers.cc:485] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 -> Peer a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69:38267): Couldn't send request to peer a3bcf2c9cfd24f25964eea216f017565. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a64ccf8bf160452c98d7cf42884eadc5. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:04.234297 24203 raft_consensus.cc:1261] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 01:54:04.234887 24781 consensus_queue.cc:1035] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:54:04.235239 24847 consensus_queue.cc:1035] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:54:04.236968 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edcd38b8d8643ea896111553e49c16c" candidate_uuid: "a3bcf2c9cfd24f25964eea216f017565" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:54:04.237210 24207 raft_consensus.cc:3037] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:04.238732 24847 raft_consensus.cc:2932] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
I20240502 01:54:04.239101 24203 raft_consensus.cc:2932] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
I20240502 01:54:04.240892 24207 raft_consensus.cc:2454] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3bcf2c9cfd24f25964eea216f017565 in term 3.
I20240502 01:54:04.241533 24572 leader_election.cc:302] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9716594f476e436b91b93ceaed4bcf59, a3bcf2c9cfd24f25964eea216f017565; no voters:
W20240502 01:54:04.243964 24076 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20240502 01:54:04.244632 24864 raft_consensus.cc:2781] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:04.245317 24022 catalog_manager.cc:5484] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: config changed from index -1 to 27, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New cstate: current_term: 1 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
W20240502 01:54:04.247069 24076 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 68d83f47ca7c4a5caeb4ffda5ebc9757, TS a6f86494e4de4d3c851c5dab07b40568 -> TS cabbb33d9a564f7ca7c2d5cceeedc477 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
W20240502 01:54:04.247159 24076 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 68d83f47ca7c4a5caeb4ffda5ebc9757, TS a6f86494e4de4d3c851c5dab07b40568 -> TS cabbb33d9a564f7ca7c2d5cceeedc477 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:04.248874 24124 raft_consensus.cc:2932] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
W20240502 01:54:04.251060 24076 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 73d4415a5e104edcbfc621ab5604c431, TS 9716594f476e436b91b93ceaed4bcf59 -> TS 03b70cf3fe3b4a32ad200d00c0c88367 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
W20240502 01:54:04.251139 24076 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 73d4415a5e104edcbfc621ab5604c431, TS 9716594f476e436b91b93ceaed4bcf59 -> TS 03b70cf3fe3b4a32ad200d00c0c88367 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:04.252851 24274 raft_consensus.cc:2454] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3bcf2c9cfd24f25964eea216f017565 in term 3.
I20240502 01:54:04.254961 24864 raft_consensus.cc:686] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [term 3 LEADER]: Becoming Leader. State: Replica: a3bcf2c9cfd24f25964eea216f017565, State: Running, Role: LEADER
W20240502 01:54:04.255323 24076 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 776becbedf8a4b89a59ea06cf8e2163a, TS 227af58cb9554c438ae592baff6af4fb -> TS cabbb33d9a564f7ca7c2d5cceeedc477 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:04.255313 24864 consensus_queue.cc:237] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
W20240502 01:54:04.255419 24076 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 776becbedf8a4b89a59ea06cf8e2163a, TS 227af58cb9554c438ae592baff6af4fb -> TS cabbb33d9a564f7ca7c2d5cceeedc477 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:04.257397 24019 catalog_manager.cc:5484] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 reported cstate change: term changed from 2 to 3, leader changed from a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) to a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69). New cstate: current_term: 3 leader_uuid: "a3bcf2c9cfd24f25964eea216f017565" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:54:04.257800 24076 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 2cf1c78d5f3f406d9029fe29deb6412a, TS 9716594f476e436b91b93ceaed4bcf59 -> TS 03b70cf3fe3b4a32ad200d00c0c88367 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
W20240502 01:54:04.257861 24076 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 2cf1c78d5f3f406d9029fe29deb6412a, TS 9716594f476e436b91b93ceaed4bcf59 -> TS 03b70cf3fe3b4a32ad200d00c0c88367 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
W20240502 01:54:04.260231 24076 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet dc63402f38f94f50b33661f6d776f813, TS a6f86494e4de4d3c851c5dab07b40568 -> TS a3bcf2c9cfd24f25964eea216f017565 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } }
W20240502 01:54:04.260296 24076 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet dc63402f38f94f50b33661f6d776f813, TS a6f86494e4de4d3c851c5dab07b40568 -> TS a3bcf2c9cfd24f25964eea216f017565 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } }
I20240502 01:54:04.500051 24203 raft_consensus.cc:1261] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Refusing update from remote peer cabbb33d9a564f7ca7c2d5cceeedc477: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 01:54:04.500541 24852 consensus_queue.cc:1035] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:04.503122 24207 raft_consensus.cc:1261] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 01:54:04.503652 24842 consensus_queue.cc:1035] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:04.519588 24129 raft_consensus.cc:1261] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Refusing update from remote peer cabbb33d9a564f7ca7c2d5cceeedc477: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 01:54:04.520342 24852 consensus_queue.cc:1035] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:04.531235 24281 raft_consensus.cc:1261] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 01:54:04.532104 24842 consensus_queue.cc:1035] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:04.593789 24281 raft_consensus.cc:1261] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 01:54:04.594326 24842 consensus_queue.cc:1035] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:04.595237 24207 raft_consensus.cc:1261] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 01:54:04.595736 24803 consensus_queue.cc:1035] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:04.599951 24682 raft_consensus.cc:1261] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 2 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 01:54:04.601859 24875 consensus_queue.cc:1035] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:04.603502 24281 raft_consensus.cc:1261] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 01:54:04.604656 24842 consensus_queue.cc:1035] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:04.607393 24207 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Refusing update from remote peer 03b70cf3fe3b4a32ad200d00c0c88367: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 01:54:04.607896 24855 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:04.619417 24281 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Refusing update from remote peer 03b70cf3fe3b4a32ad200d00c0c88367: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 01:54:04.619709 24884 ts_tablet_manager.cc:902] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565: Initiating tablet copy from peer a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:54:04.620438 24855 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:04.636291 24207 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Refusing update from remote peer a3bcf2c9cfd24f25964eea216f017565: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 01:54:04.636850 24864 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:04.641923 24884 tablet_copy_client.cc:308] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Beginning tablet copy session from remote peer at address 127.20.65.67:33253
I20240502 01:54:04.642822 24291 tablet_copy_service.cc:140] P a6f86494e4de4d3c851c5dab07b40568: Received BeginTabletCopySession request for tablet a64ccf8bf160452c98d7cf42884eadc5 from peer a3bcf2c9cfd24f25964eea216f017565 ({username='slave'} at 127.0.0.1:55132)
I20240502 01:54:04.642973 24291 tablet_copy_service.cc:161] P a6f86494e4de4d3c851c5dab07b40568: Beginning new tablet copy session on tablet a64ccf8bf160452c98d7cf42884eadc5 from peer a3bcf2c9cfd24f25964eea216f017565 at {username='slave'} at 127.0.0.1:55132: session id = a3bcf2c9cfd24f25964eea216f017565-a64ccf8bf160452c98d7cf42884eadc5
I20240502 01:54:04.646559 24291 tablet_copy_source_session.cc:205] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:04.646821 24274 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Refusing update from remote peer a3bcf2c9cfd24f25964eea216f017565: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 01:54:04.648066 24884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a64ccf8bf160452c98d7cf42884eadc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.650492 24865 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:04.655784 24884 tablet_copy_client.cc:783] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:04.656054 24884 tablet_copy_client.cc:647] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:04.657699 24890 tablet_copy_client.cc:1002] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Injecting latency into file download: 2000
I20240502 01:54:05.771265 24081 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:54:05.774876 24129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
mode: GRACEFUL
new_leader_uuid: "9716594f476e436b91b93ceaed4bcf59"
from {username='slave'} at 127.0.0.1:45086
I20240502 01:54:05.775105 24129 raft_consensus.cc:595] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Received request to transfer leadership to TS 9716594f476e436b91b93ceaed4bcf59
I20240502 01:54:05.776417 24129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "68d83f47ca7c4a5caeb4ffda5ebc9757"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
mode: GRACEFUL
new_leader_uuid: "9716594f476e436b91b93ceaed4bcf59"
from {username='slave'} at 127.0.0.1:45086
I20240502 01:54:05.776593 24129 raft_consensus.cc:595] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Received request to transfer leadership to TS 9716594f476e436b91b93ceaed4bcf59
I20240502 01:54:05.776847 24803 raft_consensus.cc:979] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb: : Instructing follower 9716594f476e436b91b93ceaed4bcf59 to start an election
I20240502 01:54:05.776964 24803 raft_consensus.cc:1067] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Signalling peer 9716594f476e436b91b93ceaed4bcf59 to start an election
I20240502 01:54:05.777494 24207 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
from {username='slave'} at 127.0.0.1:60992
I20240502 01:54:05.777675 24207 raft_consensus.cc:482] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:05.777781 24207 raft_consensus.cc:3037] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:05.777824 24129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73d4415a5e104edcbfc621ab5604c431"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
mode: GRACEFUL
new_leader_uuid: "a6f86494e4de4d3c851c5dab07b40568"
from {username='slave'} at 127.0.0.1:45086
I20240502 01:54:05.777997 24129 raft_consensus.cc:595] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Received request to transfer leadership to TS a6f86494e4de4d3c851c5dab07b40568
I20240502 01:54:05.778381 24081 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 01:54:05.778611 24081 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:54:05.780414 24207 raft_consensus.cc:504] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:05.781198 24207 leader_election.cc:288] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 2 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69:38267)
I20240502 01:54:05.781443 24129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:54:05.781611 24129 raft_consensus.cc:3032] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:05.781693 24129 raft_consensus.cc:729] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:05.781987 24129 consensus_queue.cc:260] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:05.782260 24129 raft_consensus.cc:3037] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:05.784335 24129 raft_consensus.cc:2454] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 2.
I20240502 01:54:05.785070 24171 leader_election.cc:302] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, 9716594f476e436b91b93ceaed4bcf59; no voters:
I20240502 01:54:05.785364 24841 raft_consensus.cc:2781] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:05.793629 24841 raft_consensus.cc:686] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 2 LEADER]: Becoming Leader. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:54:05.794274 24841 consensus_queue.cc:237] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:05.794929 24607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "a3bcf2c9cfd24f25964eea216f017565"
I20240502 01:54:05.795154 24607 raft_consensus.cc:3037] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:05.797905 24607 raft_consensus.cc:2454] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 2.
I20240502 01:54:05.800601 24019 catalog_manager.cc:5484] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: term changed from 1 to 2, leader changed from 227af58cb9554c438ae592baff6af4fb (127.20.65.65) to 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66). New cstate: current_term: 2 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:05.998450 24803 raft_consensus.cc:979] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb: : Instructing follower 9716594f476e436b91b93ceaed4bcf59 to start an election
I20240502 01:54:05.998627 24803 raft_consensus.cc:1067] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Signalling peer 9716594f476e436b91b93ceaed4bcf59 to start an election
I20240502 01:54:05.999228 24207 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68d83f47ca7c4a5caeb4ffda5ebc9757"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
from {username='slave'} at 127.0.0.1:60992
I20240502 01:54:05.999389 24207 raft_consensus.cc:482] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:05.999500 24207 raft_consensus.cc:3037] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:06.001705 24207 raft_consensus.cc:504] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:06.002218 24207 leader_election.cc:288] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 3 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:54:06.002545 24129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d83f47ca7c4a5caeb4ffda5ebc9757" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:54:06.002749 24129 raft_consensus.cc:3032] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:06.002818 24129 raft_consensus.cc:729] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:06.002985 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d83f47ca7c4a5caeb4ffda5ebc9757" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:54:06.003122 24281 raft_consensus.cc:3037] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:06.003037 24129 consensus_queue.cc:260] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:06.003347 24129 raft_consensus.cc:3037] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:06.006547 24129 raft_consensus.cc:2454] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 3.
I20240502 01:54:06.007098 24171 leader_election.cc:302] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, 9716594f476e436b91b93ceaed4bcf59; no voters:
I20240502 01:54:06.007532 24841 raft_consensus.cc:2781] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:06.007750 24841 raft_consensus.cc:686] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Becoming Leader. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:54:06.008049 24841 consensus_queue.cc:237] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:06.012189 24019 catalog_manager.cc:5484] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: term changed from 2 to 3, leader changed from 227af58cb9554c438ae592baff6af4fb (127.20.65.65) to 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66). New cstate: current_term: 3 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:06.025482 24281 raft_consensus.cc:2454] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 3.
I20240502 01:54:06.233834 24607 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 2 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 01:54:06.234419 24841 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:06.238392 24129 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 01:54:06.238924 24903 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:06.246366 24803 raft_consensus.cc:979] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb: : Instructing follower a6f86494e4de4d3c851c5dab07b40568 to start an election
I20240502 01:54:06.246510 24803 raft_consensus.cc:1067] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Signalling peer a6f86494e4de4d3c851c5dab07b40568 to start an election
I20240502 01:54:06.247097 24281 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73d4415a5e104edcbfc621ab5604c431"
dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
from {username='slave'} at 127.0.0.1:55086
I20240502 01:54:06.247259 24281 raft_consensus.cc:482] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:06.247357 24281 raft_consensus.cc:3037] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:06.249760 24281 raft_consensus.cc:504] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:06.250317 24281 leader_election.cc:288] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 3 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:54:06.250636 24129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d4415a5e104edcbfc621ab5604c431" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:54:06.250725 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d4415a5e104edcbfc621ab5604c431" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:54:06.250799 24129 raft_consensus.cc:3032] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:06.250896 24129 raft_consensus.cc:729] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:06.250913 24207 raft_consensus.cc:3037] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:06.251116 24129 consensus_queue.cc:260] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:06.251375 24129 raft_consensus.cc:3037] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:06.253213 24207 raft_consensus.cc:2454] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 3.
I20240502 01:54:06.253821 24129 raft_consensus.cc:2454] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 3.
I20240502 01:54:06.254259 24246 leader_election.cc:302] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:54:06.254724 24900 raft_consensus.cc:2781] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:06.255148 24900 raft_consensus.cc:686] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [term 3 LEADER]: Becoming Leader. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Running, Role: LEADER
I20240502 01:54:06.255420 24900 consensus_queue.cc:237] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:06.258028 24019 catalog_manager.cc:5484] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: term changed from 2 to 3, leader changed from 227af58cb9554c438ae592baff6af4fb (127.20.65.65) to a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67). New cstate: current_term: 3 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:06.534279 24281 raft_consensus.cc:1261] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 01:54:06.534826 24903 consensus_queue.cc:1035] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:06.538262 24129 raft_consensus.cc:1261] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 01:54:06.538756 24903 consensus_queue.cc:1035] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:06.658046 24884 tablet_copy_client.cc:523] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:06.660694 24884 tablet_bootstrap.cc:492] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565: Bootstrap starting.
I20240502 01:54:06.690382 24884 tablet_bootstrap.cc:492] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:06.690802 24884 tablet_bootstrap.cc:492] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565: Bootstrap complete.
I20240502 01:54:06.691083 24884 ts_tablet_manager.cc:1367] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.007s
I20240502 01:54:06.691727 24884 raft_consensus.cc:348] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:06.692004 24884 raft_consensus.cc:729] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a3bcf2c9cfd24f25964eea216f017565, State: Initialized, Role: LEARNER
I20240502 01:54:06.692214 24884 consensus_queue.cc:260] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:06.692906 24884 ts_tablet_manager.cc:1398] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:06.693605 24291 tablet_copy_service.cc:342] P a6f86494e4de4d3c851c5dab07b40568: Request end of tablet copy session a3bcf2c9cfd24f25964eea216f017565-a64ccf8bf160452c98d7cf42884eadc5 received from {username='slave'} at 127.0.0.1:55132
I20240502 01:54:06.693715 24291 tablet_copy_service.cc:433] P a6f86494e4de4d3c851c5dab07b40568: ending tablet copy session a3bcf2c9cfd24f25964eea216f017565-a64ccf8bf160452c98d7cf42884eadc5 on tablet a64ccf8bf160452c98d7cf42884eadc5 with peer a3bcf2c9cfd24f25964eea216f017565
I20240502 01:54:06.751148 24207 raft_consensus.cc:1261] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 01:54:06.751631 24900 consensus_queue.cc:1035] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:06.755728 24129 raft_consensus.cc:1261] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 01:54:06.756259 24916 consensus_queue.cc:1035] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:06.879969 24607 raft_consensus.cc:1203] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240502 01:54:07.470734 24930 raft_consensus.cc:1050] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568: attempting to promote NON_VOTER a3bcf2c9cfd24f25964eea216f017565 to VOTER
I20240502 01:54:07.471638 24930 consensus_queue.cc:237] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:07.473973 24607 raft_consensus.cc:1261] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565 [term 1 LEARNER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:54:07.474114 24207 raft_consensus.cc:1261] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:54:07.474251 24129 raft_consensus.cc:1261] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:54:07.474925 24916 consensus_queue.cc:1035] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:07.475242 24929 consensus_queue.cc:1035] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:07.475847 24916 consensus_queue.cc:1035] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:07.484097 24916 raft_consensus.cc:2932] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:07.484872 24129 raft_consensus.cc:2932] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:07.485226 24607 raft_consensus.cc:2932] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:07.488032 24207 raft_consensus.cc:2932] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:07.491106 24022 catalog_manager.cc:5484] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565 reported cstate change: config changed from index 27 to 28, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:07.498919 24281 consensus_queue.cc:237] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:07.500571 24207 raft_consensus.cc:1261] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 01:54:07.500571 24607 raft_consensus.cc:1261] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565 [term 1 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 01:54:07.501096 24930 consensus_queue.cc:1035] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:07.501358 24932 consensus_queue.cc:1035] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:07.503450 24930 raft_consensus.cc:2932] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 227af58cb9554c438ae592baff6af4fb (127.20.65.65) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:07.504040 24207 raft_consensus.cc:2932] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 227af58cb9554c438ae592baff6af4fb (127.20.65.65) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:07.507522 24015 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a64ccf8bf160452c98d7cf42884eadc5 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:07.507431 24607 raft_consensus.cc:2932] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 227af58cb9554c438ae592baff6af4fb (127.20.65.65) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:07.507874 24027 catalog_manager.cc:5484] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: config changed from index 28 to 29, VOTER 227af58cb9554c438ae592baff6af4fb (127.20.65.65) evicted. New cstate: current_term: 1 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:07.511411 24120 tablet_service.cc:1514] Processing DeleteTablet for tablet a64ccf8bf160452c98d7cf42884eadc5 with delete_type TABLET_DATA_TOMBSTONED (TS 227af58cb9554c438ae592baff6af4fb not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:45056
I20240502 01:54:07.512126 24940 tablet_replica.cc:330] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb: stopping tablet replica
I20240502 01:54:07.512357 24940 raft_consensus.cc:2229] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:07.512670 24940 raft_consensus.cc:2258] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.513550 24940 ts_tablet_manager.cc:1875] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:07.518160 24940 ts_tablet_manager.cc:1888] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240502 01:54:07.518244 24940 log.cc:1196] T a64ccf8bf160452c98d7cf42884eadc5 P 227af58cb9554c438ae592baff6af4fb: Deleting WAL directory at /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-0-root/wals/a64ccf8bf160452c98d7cf42884eadc5
I20240502 01:54:07.518898 24014 catalog_manager.cc:4830] TS 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551): tablet a64ccf8bf160452c98d7cf42884eadc5 (table test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]) successfully deleted
I20240502 01:54:07.778936 24081 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:54:07.781478 24207 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
mode: GRACEFUL
new_leader_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:32776
I20240502 01:54:07.781659 24207 raft_consensus.cc:595] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 2 LEADER]: Received request to transfer leadership to TS 227af58cb9554c438ae592baff6af4fb
I20240502 01:54:07.782703 24274 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a64ccf8bf160452c98d7cf42884eadc5"
dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
mode: GRACEFUL
new_leader_uuid: "9716594f476e436b91b93ceaed4bcf59"
from {username='slave'} at 127.0.0.1:55106
I20240502 01:54:07.782874 24274 raft_consensus.cc:595] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Received request to transfer leadership to TS 9716594f476e436b91b93ceaed4bcf59
I20240502 01:54:07.783257 24942 raft_consensus.cc:979] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59: : Instructing follower 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:54:07.783388 24942 raft_consensus.cc:1067] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 2 LEADER]: Signalling peer 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:54:07.783913 24207 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0684f060716413796e90d5439d5d482"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
mode: GRACEFUL
new_leader_uuid: "a6f86494e4de4d3c851c5dab07b40568"
from {username='slave'} at 127.0.0.1:32776
I20240502 01:54:07.783942 24129 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:45068
I20240502 01:54:07.784085 24207 raft_consensus.cc:595] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Received request to transfer leadership to TS a6f86494e4de4d3c851c5dab07b40568
I20240502 01:54:07.784090 24129 raft_consensus.cc:482] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:07.784191 24129 raft_consensus.cc:3037] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:07.784478 24081 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 01:54:07.784694 24081 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:54:07.786202 24129 raft_consensus.cc:504] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:07.786720 24129 leader_election.cc:288] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 3 election: Requested vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69:38267)
I20240502 01:54:07.787135 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:54:07.787279 24207 raft_consensus.cc:3032] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:07.787235 24607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "a3bcf2c9cfd24f25964eea216f017565"
I20240502 01:54:07.787358 24207 raft_consensus.cc:729] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:54:07.787416 24607 raft_consensus.cc:3037] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:07.787590 24207 consensus_queue.cc:260] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:07.787881 24207 raft_consensus.cc:3037] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:07.789273 24607 raft_consensus.cc:2454] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 3.
I20240502 01:54:07.789654 24095 leader_election.cc:302] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, a3bcf2c9cfd24f25964eea216f017565; no voters:
I20240502 01:54:07.789839 24207 raft_consensus.cc:2454] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 3.
I20240502 01:54:07.789897 24842 raft_consensus.cc:2781] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:07.790225 24842 raft_consensus.cc:686] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 3 LEADER]: Becoming Leader. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:07.790520 24842 consensus_queue.cc:237] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:07.793114 24026 catalog_manager.cc:5484] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb reported cstate change: term changed from 2 to 3, leader changed from 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) to 227af58cb9554c438ae592baff6af4fb (127.20.65.65). New cstate: current_term: 3 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:07.954185 24903 raft_consensus.cc:979] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59: : Instructing follower a6f86494e4de4d3c851c5dab07b40568 to start an election
I20240502 01:54:07.954324 24903 raft_consensus.cc:1067] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Signalling peer a6f86494e4de4d3c851c5dab07b40568 to start an election
I20240502 01:54:07.954949 24274 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0684f060716413796e90d5439d5d482"
dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
from {username='slave'} at 127.0.0.1:55076
I20240502 01:54:07.955137 24274 raft_consensus.cc:482] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:07.955229 24274 raft_consensus.cc:3037] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:07.957137 24274 raft_consensus.cc:504] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:07.957680 24274 leader_election.cc:288] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 2 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:54:07.958060 24129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0684f060716413796e90d5439d5d482" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:54:07.958238 24129 raft_consensus.cc:3037] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:07.958182 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0684f060716413796e90d5439d5d482" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:54:07.958379 24207 raft_consensus.cc:3032] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:07.958464 24207 raft_consensus.cc:729] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:54:07.958732 24207 consensus_queue.cc:260] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:07.959019 24207 raft_consensus.cc:3037] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:07.960350 24129 raft_consensus.cc:2454] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 2.
I20240502 01:54:07.960796 24246 leader_election.cc:302] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:54:07.961006 24207 raft_consensus.cc:2454] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 2.
I20240502 01:54:07.961130 24932 raft_consensus.cc:2781] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:07.961282 24932 raft_consensus.cc:686] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Becoming Leader. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Running, Role: LEADER
I20240502 01:54:07.961758 24932 consensus_queue.cc:237] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } }
I20240502 01:54:07.964375 24026 catalog_manager.cc:5484] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: term changed from 1 to 2, leader changed from 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) to a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67). New cstate: current_term: 2 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:08.072242 24930 raft_consensus.cc:979] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568: : Instructing follower 9716594f476e436b91b93ceaed4bcf59 to start an election
I20240502 01:54:08.072404 24930 raft_consensus.cc:1067] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Signalling peer 9716594f476e436b91b93ceaed4bcf59 to start an election
I20240502 01:54:08.072938 24207 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a64ccf8bf160452c98d7cf42884eadc5"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
from {username='slave'} at 127.0.0.1:60982
I20240502 01:54:08.073113 24207 raft_consensus.cc:482] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:08.073204 24207 raft_consensus.cc:3037] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:08.075206 24207 raft_consensus.cc:504] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:08.075752 24207 leader_election.cc:288] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 2 election: Requested vote from peers a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253), a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69:38267)
I20240502 01:54:08.076175 24274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64ccf8bf160452c98d7cf42884eadc5" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:54:08.076210 24607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64ccf8bf160452c98d7cf42884eadc5" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "a3bcf2c9cfd24f25964eea216f017565"
I20240502 01:54:08.076391 24274 raft_consensus.cc:3032] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:08.076434 24607 raft_consensus.cc:3037] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:08.076459 24274 raft_consensus.cc:729] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Running, Role: LEADER
I20240502 01:54:08.076691 24274 consensus_queue.cc:260] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:08.076969 24274 raft_consensus.cc:3037] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:08.078374 24607 raft_consensus.cc:2454] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 2.
I20240502 01:54:08.078773 24171 leader_election.cc:302] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9716594f476e436b91b93ceaed4bcf59, a3bcf2c9cfd24f25964eea216f017565; no voters:
I20240502 01:54:08.078928 24274 raft_consensus.cc:2454] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 2.
I20240502 01:54:08.079000 24903 raft_consensus.cc:2781] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:08.079103 24903 raft_consensus.cc:686] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 2 LEADER]: Becoming Leader. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:54:08.079336 24903 consensus_queue.cc:237] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:08.081378 24026 catalog_manager.cc:5484] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: term changed from 1 to 2, leader changed from a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) to 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66). New cstate: current_term: 2 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:08.305390 24207 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 01:54:08.305980 24842 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 01:54:08.307657 24607 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 3 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 01:54:08.308189 24842 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 01:54:08.378123 24207 raft_consensus.cc:1261] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 01:54:08.378671 24930 consensus_queue.cc:1035] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:54:08.382798 24124 raft_consensus.cc:1261] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 01:54:08.383491 24932 consensus_queue.cc:1035] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:54:08.513665 24207 consensus_queue.cc:237] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } }
I20240502 01:54:08.515913 24274 raft_consensus.cc:1261] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 01:54:08.515913 24124 raft_consensus.cc:1261] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
W20240502 01:54:08.516155 24171 consensus_peers.cc:485] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 -> Peer 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70:35815): Couldn't send request to peer 03b70cf3fe3b4a32ad200d00c0c88367. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68d83f47ca7c4a5caeb4ffda5ebc9757. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:08.516502 24959 consensus_queue.cc:1035] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:08.516827 24903 consensus_queue.cc:1035] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:08.519598 24942 raft_consensus.cc:2932] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:54:08.520195 24124 raft_consensus.cc:2932] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:54:08.520200 24274 raft_consensus.cc:2932] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:54:08.526856 24026 catalog_manager.cc:5484] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb reported cstate change: config changed from index -1 to 30, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New cstate: current_term: 3 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:54:08.527958 24274 consensus_queue.cc:237] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:08.531594 24207 raft_consensus.cc:1261] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 01:54:08.532357 24124 raft_consensus.cc:1261] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 01:54:08.532560 24955 consensus_queue.cc:1035] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:54:08.532912 24932 consensus_queue.cc:1035] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:54:08.535326 24955 raft_consensus.cc:2932] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
I20240502 01:54:08.536015 24207 raft_consensus.cc:2932] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
I20240502 01:54:08.539855 24027 catalog_manager.cc:5484] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: config changed from index -1 to 27, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New cstate: current_term: 2 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:08.543294 24124 raft_consensus.cc:2932] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
W20240502 01:54:08.548599 24246 consensus_peers.cc:485] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 -> Peer cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68:38415): Couldn't send request to peer cabbb33d9a564f7ca7c2d5cceeedc477. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0684f060716413796e90d5439d5d482. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:08.561594 24607 consensus_queue.cc:237] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } }
I20240502 01:54:08.564096 24274 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Refusing update from remote peer a3bcf2c9cfd24f25964eea216f017565: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 01:54:08.564819 24207 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Refusing update from remote peer a3bcf2c9cfd24f25964eea216f017565: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 01:54:08.564783 24897 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 01:54:08.565371 24967 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 01:54:08.567832 24897 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 30 to 32, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:54:08.569504 24207 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 30 to 32, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:54:08.571655 24274 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 30 to 32, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:54:08.572690 24027 catalog_manager.cc:5484] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 reported cstate change: config changed from index 30 to 32, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New cstate: current_term: 3 leader_uuid: "a3bcf2c9cfd24f25964eea216f017565" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:08.573158 24203 consensus_queue.cc:237] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
W20240502 01:54:08.575675 24572 consensus_peers.cc:485] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 -> Peer 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70:35815): Couldn't send request to peer 03b70cf3fe3b4a32ad200d00c0c88367. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1edcd38b8d8643ea896111553e49c16c. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:08.579159 24171 consensus_peers.cc:485] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 -> Peer a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69:38267): Couldn't send request to peer a3bcf2c9cfd24f25964eea216f017565. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92d3bd85ba5a4586b9f3ef23824c0287. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:08.579285 24274 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 01:54:08.579761 24124 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 01:54:08.579851 24942 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:08.580201 24903 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:08.582490 24942 raft_consensus.cc:2932] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index -1 to 29, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
I20240502 01:54:08.583063 24274 raft_consensus.cc:2932] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index -1 to 29, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
I20240502 01:54:08.586562 24124 raft_consensus.cc:2932] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index -1 to 29, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
I20240502 01:54:08.588001 24026 catalog_manager.cc:5484] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: config changed from index -1 to 29, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New cstate: current_term: 3 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:08.596632 24682 consensus_queue.cc:237] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:08.599201 24274 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Refusing update from remote peer 03b70cf3fe3b4a32ad200d00c0c88367: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 01:54:08.599690 24976 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 01:54:08.602069 24941 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
I20240502 01:54:08.602648 24274 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
I20240502 01:54:08.606127 24203 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Refusing update from remote peer 03b70cf3fe3b4a32ad200d00c0c88367: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 01:54:08.607234 24976 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 01:54:08.607928 24019 catalog_manager.cc:5484] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: config changed from index 30 to 32, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New cstate: current_term: 2 leader_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
W20240502 01:54:08.610615 24076 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240502 01:54:08.615851 24647 consensus_peers.cc:485] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 -> Peer cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68:38415): Couldn't send request to peer cabbb33d9a564f7ca7c2d5cceeedc477. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d516fd48dce443169e7e6011ea6a704d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:08.616478 24203 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
W20240502 01:54:08.619397 24076 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet a64ccf8bf160452c98d7cf42884eadc5, TS a6f86494e4de4d3c851c5dab07b40568 -> TS cabbb33d9a564f7ca7c2d5cceeedc477 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
W20240502 01:54:08.619474 24076 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet a64ccf8bf160452c98d7cf42884eadc5, TS a6f86494e4de4d3c851c5dab07b40568 -> TS cabbb33d9a564f7ca7c2d5cceeedc477 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:08.640110 24274 raft_consensus.cc:1261] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 01:54:08.640520 24959 consensus_queue.cc:1035] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:08.644555 24608 raft_consensus.cc:1261] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565 [term 2 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 01:54:08.645054 24959 consensus_queue.cc:1035] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:08.995447 24984 ts_tablet_manager.cc:902] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367: Initiating tablet copy from peer 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:54:08.997267 24984 tablet_copy_client.cc:308] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Beginning tablet copy session from remote peer at address 127.20.65.66:37365
I20240502 01:54:08.997860 24217 tablet_copy_service.cc:140] P 9716594f476e436b91b93ceaed4bcf59: Received BeginTabletCopySession request for tablet 68d83f47ca7c4a5caeb4ffda5ebc9757 from peer 03b70cf3fe3b4a32ad200d00c0c88367 ({username='slave'} at 127.0.0.1:32800)
I20240502 01:54:08.998003 24217 tablet_copy_service.cc:161] P 9716594f476e436b91b93ceaed4bcf59: Beginning new tablet copy session on tablet 68d83f47ca7c4a5caeb4ffda5ebc9757 from peer 03b70cf3fe3b4a32ad200d00c0c88367 at {username='slave'} at 127.0.0.1:32800: session id = 03b70cf3fe3b4a32ad200d00c0c88367-68d83f47ca7c4a5caeb4ffda5ebc9757
I20240502 01:54:09.000692 24217 tablet_copy_source_session.cc:205] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.001875 24984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68d83f47ca7c4a5caeb4ffda5ebc9757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.006917 24984 tablet_copy_client.cc:783] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.007169 24984 tablet_copy_client.cc:647] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.008423 24988 tablet_copy_client.cc:1002] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Injecting latency into file download: 2000
I20240502 01:54:09.043033 24989 ts_tablet_manager.cc:902] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477: Initiating tablet copy from peer a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:54:09.051784 24989 tablet_copy_client.cc:308] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Beginning tablet copy session from remote peer at address 127.20.65.67:33253
I20240502 01:54:09.052507 24995 ts_tablet_manager.cc:902] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565: Initiating tablet copy from peer 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:54:09.062593 24993 ts_tablet_manager.cc:902] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477: Initiating tablet copy from peer 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70:35815)
I20240502 01:54:09.066844 24291 tablet_copy_service.cc:140] P a6f86494e4de4d3c851c5dab07b40568: Received BeginTabletCopySession request for tablet f0684f060716413796e90d5439d5d482 from peer cabbb33d9a564f7ca7c2d5cceeedc477 ({username='slave'} at 127.0.0.1:58482)
I20240502 01:54:09.066983 24291 tablet_copy_service.cc:161] P a6f86494e4de4d3c851c5dab07b40568: Beginning new tablet copy session on tablet f0684f060716413796e90d5439d5d482 from peer cabbb33d9a564f7ca7c2d5cceeedc477 at {username='slave'} at 127.0.0.1:58482: session id = cabbb33d9a564f7ca7c2d5cceeedc477-f0684f060716413796e90d5439d5d482
I20240502 01:54:09.069212 24291 tablet_copy_source_session.cc:205] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.070550 24989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0684f060716413796e90d5439d5d482. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.072315 24993 tablet_copy_client.cc:308] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Beginning tablet copy session from remote peer at address 127.20.65.70:35815
I20240502 01:54:09.075697 24995 tablet_copy_client.cc:308] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Beginning tablet copy session from remote peer at address 127.20.65.66:37365
I20240502 01:54:09.076237 24217 tablet_copy_service.cc:140] P 9716594f476e436b91b93ceaed4bcf59: Received BeginTabletCopySession request for tablet 92d3bd85ba5a4586b9f3ef23824c0287 from peer a3bcf2c9cfd24f25964eea216f017565 ({username='slave'} at 127.0.0.1:32816)
I20240502 01:54:09.076349 24217 tablet_copy_service.cc:161] P 9716594f476e436b91b93ceaed4bcf59: Beginning new tablet copy session on tablet 92d3bd85ba5a4586b9f3ef23824c0287 from peer a3bcf2c9cfd24f25964eea216f017565 at {username='slave'} at 127.0.0.1:32816: session id = a3bcf2c9cfd24f25964eea216f017565-92d3bd85ba5a4586b9f3ef23824c0287
I20240502 01:54:09.079258 24217 tablet_copy_source_session.cc:205] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.079813 24989 tablet_copy_client.cc:783] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.080073 24989 tablet_copy_client.cc:647] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.081809 24995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d3bd85ba5a4586b9f3ef23824c0287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.082067 24692 tablet_copy_service.cc:140] P 03b70cf3fe3b4a32ad200d00c0c88367: Received BeginTabletCopySession request for tablet d516fd48dce443169e7e6011ea6a704d from peer cabbb33d9a564f7ca7c2d5cceeedc477 ({username='slave'} at 127.0.0.1:42956)
I20240502 01:54:09.082187 24692 tablet_copy_service.cc:161] P 03b70cf3fe3b4a32ad200d00c0c88367: Beginning new tablet copy session on tablet d516fd48dce443169e7e6011ea6a704d from peer cabbb33d9a564f7ca7c2d5cceeedc477 at {username='slave'} at 127.0.0.1:42956: session id = cabbb33d9a564f7ca7c2d5cceeedc477-d516fd48dce443169e7e6011ea6a704d
I20240502 01:54:09.082768 24994 tablet_copy_client.cc:1002] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Injecting latency into file download: 2000
I20240502 01:54:09.085299 24692 tablet_copy_source_session.cc:205] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.086357 24993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d516fd48dce443169e7e6011ea6a704d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.088282 24995 tablet_copy_client.cc:783] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.088516 24995 tablet_copy_client.cc:647] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.089761 25005 tablet_copy_client.cc:1002] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Injecting latency into file download: 2000
I20240502 01:54:09.091938 24993 tablet_copy_client.cc:783] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.092157 24993 tablet_copy_client.cc:647] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.093463 25004 tablet_copy_client.cc:1002] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Injecting latency into file download: 2000
I20240502 01:54:09.125608 25006 ts_tablet_manager.cc:902] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367: Initiating tablet copy from peer a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69:38267)
I20240502 01:54:09.126888 25006 tablet_copy_client.cc:308] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Beginning tablet copy session from remote peer at address 127.20.65.69:38267
I20240502 01:54:09.132906 24618 tablet_copy_service.cc:140] P a3bcf2c9cfd24f25964eea216f017565: Received BeginTabletCopySession request for tablet 1edcd38b8d8643ea896111553e49c16c from peer 03b70cf3fe3b4a32ad200d00c0c88367 ({username='slave'} at 127.0.0.1:39962)
I20240502 01:54:09.133061 24618 tablet_copy_service.cc:161] P a3bcf2c9cfd24f25964eea216f017565: Beginning new tablet copy session on tablet 1edcd38b8d8643ea896111553e49c16c from peer 03b70cf3fe3b4a32ad200d00c0c88367 at {username='slave'} at 127.0.0.1:39962: session id = 03b70cf3fe3b4a32ad200d00c0c88367-1edcd38b8d8643ea896111553e49c16c
I20240502 01:54:09.135550 24618 tablet_copy_source_session.cc:205] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.136814 25006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edcd38b8d8643ea896111553e49c16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.142037 25006 tablet_copy_client.cc:783] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.142246 25006 tablet_copy_client.cc:647] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.143661 25010 tablet_copy_client.cc:1002] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Injecting latency into file download: 2000
I20240502 01:54:09.785053 24081 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:54:09.788791 24207 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
mode: GRACEFUL
new_leader_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:32776
I20240502 01:54:09.789027 24207 raft_consensus.cc:595] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Received request to transfer leadership to TS 227af58cb9554c438ae592baff6af4fb
I20240502 01:54:09.790033 24129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3f5d1021b8b4932b7fe13b7fb933a9c"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
mode: GRACEFUL
new_leader_uuid: "a6f86494e4de4d3c851c5dab07b40568"
from {username='slave'} at 127.0.0.1:45086
I20240502 01:54:09.790189 24129 raft_consensus.cc:595] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Received request to transfer leadership to TS a6f86494e4de4d3c851c5dab07b40568
I20240502 01:54:09.791124 24274 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0684f060716413796e90d5439d5d482"
dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
mode: GRACEFUL
new_leader_uuid: "9716594f476e436b91b93ceaed4bcf59"
from {username='slave'} at 127.0.0.1:55106
I20240502 01:54:09.791289 24274 raft_consensus.cc:595] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Received request to transfer leadership to TS 9716594f476e436b91b93ceaed4bcf59
I20240502 01:54:09.791678 24081 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 01:54:09.791896 24081 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:54:09.903920 24951 raft_consensus.cc:979] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb: : Instructing follower a6f86494e4de4d3c851c5dab07b40568 to start an election
I20240502 01:54:09.904103 25020 raft_consensus.cc:1067] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Signalling peer a6f86494e4de4d3c851c5dab07b40568 to start an election
I20240502 01:54:09.904816 24274 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3f5d1021b8b4932b7fe13b7fb933a9c"
dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
from {username='slave'} at 127.0.0.1:55086
I20240502 01:54:09.905007 24274 raft_consensus.cc:482] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:09.905121 24274 raft_consensus.cc:3037] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:09.907722 24274 raft_consensus.cc:504] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:09.908330 24274 leader_election.cc:288] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 2 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68:38415)
I20240502 01:54:09.908725 24129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f5d1021b8b4932b7fe13b7fb933a9c" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:54:09.908737 24532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f5d1021b8b4932b7fe13b7fb933a9c" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477"
I20240502 01:54:09.908917 24129 raft_consensus.cc:3032] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:09.909020 24129 raft_consensus.cc:729] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:09.908937 24532 raft_consensus.cc:3037] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:09.909246 24129 consensus_queue.cc:260] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:09.909536 24129 raft_consensus.cc:3037] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:09.911082 24532 raft_consensus.cc:2454] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 2.
I20240502 01:54:09.911507 24129 raft_consensus.cc:2454] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 2.
I20240502 01:54:09.912192 24246 leader_election.cc:302] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:54:09.912420 25018 raft_consensus.cc:2781] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:09.912549 25018 raft_consensus.cc:686] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Becoming Leader. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Running, Role: LEADER
I20240502 01:54:09.912789 25018 consensus_queue.cc:237] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:09.915756 24027 catalog_manager.cc:5484] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: term changed from 1 to 2, leader changed from 227af58cb9554c438ae592baff6af4fb (127.20.65.65) to a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67). New cstate: current_term: 2 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:10.123631 25013 raft_consensus.cc:979] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59: : Instructing follower 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:54:10.123805 24959 raft_consensus.cc:1067] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Signalling peer 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:54:10.124373 24129 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:45068
I20240502 01:54:10.124567 24129 raft_consensus.cc:482] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:10.124677 24129 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:10.127353 24129 raft_consensus.cc:504] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:10.128335 24129 leader_election.cc:288] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 4 election: Requested vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:54:10.128371 24281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:54:10.128676 24281 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:10.128341 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:54:10.129052 24207 raft_consensus.cc:3032] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:10.129150 24207 raft_consensus.cc:729] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:54:10.129418 24207 consensus_queue.cc:260] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:10.129859 24207 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:10.131336 24281 raft_consensus.cc:2454] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 4.
I20240502 01:54:10.131785 24096 leader_election.cc:302] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, a6f86494e4de4d3c851c5dab07b40568; no voters:
I20240502 01:54:10.132048 25020 raft_consensus.cc:2781] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:54:10.132167 25020 raft_consensus.cc:686] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 4 LEADER]: Becoming Leader. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:10.132452 24207 raft_consensus.cc:2454] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 4.
I20240502 01:54:10.132369 25020 consensus_queue.cc:237] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:10.135102 24027 catalog_manager.cc:5484] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb reported cstate change: term changed from 3 to 4, leader changed from 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) to 227af58cb9554c438ae592baff6af4fb (127.20.65.65). New cstate: current_term: 4 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:10.139309 24129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45056
I20240502 01:54:10.139473 24129 raft_consensus.cc:595] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 4 LEADER]: Received request to transfer leadership
I20240502 01:54:10.145146 24129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45056
I20240502 01:54:10.145279 24129 raft_consensus.cc:595] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 4 LEADER]: Received request to transfer leadership
W20240502 01:54:10.145637 24014 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 92d3bd85ba5a4586b9f3ef23824c0287 already in progress
I20240502 01:54:10.146493 24129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45056
I20240502 01:54:10.146636 24129 raft_consensus.cc:595] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 4 LEADER]: Received request to transfer leadership
W20240502 01:54:10.147073 24076 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 92d3bd85ba5a4586b9f3ef23824c0287 already in progress
W20240502 01:54:10.147135 24076 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 92d3bd85ba5a4586b9f3ef23824c0287 already in progress
I20240502 01:54:10.264155 25026 raft_consensus.cc:979] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568: : Instructing follower 9716594f476e436b91b93ceaed4bcf59 to start an election
I20240502 01:54:10.264303 25026 raft_consensus.cc:1067] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Signalling peer 9716594f476e436b91b93ceaed4bcf59 to start an election
I20240502 01:54:10.264850 24203 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0684f060716413796e90d5439d5d482"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
from {username='slave'} at 127.0.0.1:60982
I20240502 01:54:10.265035 24203 raft_consensus.cc:482] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:10.265137 24203 raft_consensus.cc:3037] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:10.267921 24203 raft_consensus.cc:504] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:10.268589 24203 leader_election.cc:288] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 3 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:54:10.268890 24129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0684f060716413796e90d5439d5d482" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:54:10.269068 24129 raft_consensus.cc:3037] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:10.270876 24129 raft_consensus.cc:2454] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 3.
I20240502 01:54:10.271174 24171 leader_election.cc:302] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, 9716594f476e436b91b93ceaed4bcf59; no voters:
I20240502 01:54:10.271212 24274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0684f060716413796e90d5439d5d482" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:54:10.271364 24274 raft_consensus.cc:3032] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:10.271440 24274 raft_consensus.cc:729] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Running, Role: LEADER
I20240502 01:54:10.271437 24959 raft_consensus.cc:2781] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:10.271590 24959 raft_consensus.cc:686] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Becoming Leader. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:54:10.271703 24274 consensus_queue.cc:260] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:10.271814 24959 consensus_queue.cc:237] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:10.271991 24274 raft_consensus.cc:3037] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:10.274570 24274 raft_consensus.cc:2454] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 3.
I20240502 01:54:10.274974 24027 catalog_manager.cc:5484] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: term changed from 2 to 3, leader changed from a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) to 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66). New cstate: current_term: 3 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:10.279588 24203 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0684f060716413796e90d5439d5d482"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60978
I20240502 01:54:10.279739 24203 raft_consensus.cc:595] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Received request to transfer leadership
I20240502 01:54:10.283996 24203 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0684f060716413796e90d5439d5d482"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60978
I20240502 01:54:10.284119 24203 raft_consensus.cc:595] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Received request to transfer leadership
I20240502 01:54:10.284850 24203 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0684f060716413796e90d5439d5d482"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60978
I20240502 01:54:10.284957 24203 raft_consensus.cc:595] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Received request to transfer leadership
W20240502 01:54:10.285297 24076 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for f0684f060716413796e90d5439d5d482 already in progress
W20240502 01:54:10.285383 24076 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for f0684f060716413796e90d5439d5d482 already in progress
I20240502 01:54:10.325068 24129 raft_consensus.cc:1261] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 01:54:10.325594 25026 consensus_queue.cc:1035] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:10.329243 24532 raft_consensus.cc:1261] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 01:54:10.330020 25018 consensus_queue.cc:1035] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240502 01:54:10.561488 24095 consensus_peers.cc:485] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb -> Peer a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69:38267): Couldn't send request to peer a3bcf2c9cfd24f25964eea216f017565. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:10.666783 24281 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 4 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240502 01:54:10.667459 25020 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:10.670225 25039 raft_consensus.cc:979] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb: : Instructing follower a6f86494e4de4d3c851c5dab07b40568 to start an election
I20240502 01:54:10.670449 25020 raft_consensus.cc:1067] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 4 LEADER]: Signalling peer a6f86494e4de4d3c851c5dab07b40568 to start an election
I20240502 01:54:10.671099 24274 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287"
dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
from {username='slave'} at 127.0.0.1:55086
I20240502 01:54:10.671253 24274 raft_consensus.cc:482] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:10.671330 24274 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:10.672202 24207 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 4 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240502 01:54:10.672991 25040 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:10.673993 24274 raft_consensus.cc:504] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:10.674976 24281 raft_consensus.cc:1226] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 5 FOLLOWER]: Rejecting Update request from peer 227af58cb9554c438ae592baff6af4fb for earlier term 4. Current term is 5. Ops: []
I20240502 01:54:10.675006 24129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:54:10.675177 24129 raft_consensus.cc:3032] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 4 LEADER]: Stepping down as leader of term 4
I20240502 01:54:10.675247 24129 raft_consensus.cc:729] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:10.675503 24129 consensus_queue.cc:260] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:10.675755 25040 consensus_queue.cc:1046] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: INVALID_TERM, Last received: 4.30, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:10.676101 24129 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:10.676824 24274 leader_election.cc:288] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 5 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
I20240502 01:54:10.677161 24203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287" candidate_uuid: "a6f86494e4de4d3c851c5dab07b40568" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:54:10.678599 24129 raft_consensus.cc:2454] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f86494e4de4d3c851c5dab07b40568 in term 5.
I20240502 01:54:10.678992 24246 leader_election.cc:302] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, a6f86494e4de4d3c851c5dab07b40568; no voters: 9716594f476e436b91b93ceaed4bcf59
I20240502 01:54:10.679370 25018 raft_consensus.cc:2781] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:10.679512 25018 raft_consensus.cc:686] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 5 LEADER]: Becoming Leader. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Running, Role: LEADER
I20240502 01:54:10.679752 25018 consensus_queue.cc:237] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.30, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:10.682032 24027 catalog_manager.cc:5484] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: term changed from 4 to 5, leader changed from 227af58cb9554c438ae592baff6af4fb (127.20.65.65) to a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67). New cstate: current_term: 5 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:54:10.752188 24171 consensus_peers.cc:485] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 -> Peer cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68:38415): Couldn't send request to peer cabbb33d9a564f7ca7c2d5cceeedc477. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:10.822870 24124 raft_consensus.cc:1261] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 01:54:10.823375 24959 consensus_queue.cc:1035] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:10.826045 25046 raft_consensus.cc:979] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59: : Instructing follower 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:54:10.826311 24959 raft_consensus.cc:1067] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Signalling peer 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:54:10.827219 24124 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0684f060716413796e90d5439d5d482"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:45068
I20240502 01:54:10.827397 24124 raft_consensus.cc:482] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:10.827476 24124 raft_consensus.cc:3037] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:10.828557 24274 raft_consensus.cc:1261] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 01:54:10.829102 25047 consensus_queue.cc:1035] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:10.831405 24124 raft_consensus.cc:504] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:10.832574 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0684f060716413796e90d5439d5d482" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:54:10.832751 24207 raft_consensus.cc:3032] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:10.832726 24274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0684f060716413796e90d5439d5d482" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:54:10.832821 24207 raft_consensus.cc:729] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:54:10.832875 24274 raft_consensus.cc:3037] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:10.833076 24207 consensus_queue.cc:260] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:10.833464 24207 raft_consensus.cc:3037] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:10.835726 24274 raft_consensus.cc:2454] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 4.
I20240502 01:54:10.836051 24207 raft_consensus.cc:2454] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 4.
I20240502 01:54:10.836334 24095 leader_election.cc:302] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, 9716594f476e436b91b93ceaed4bcf59; no voters:
I20240502 01:54:10.836621 25020 raft_consensus.cc:2781] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:54:10.836751 25020 raft_consensus.cc:686] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 4 LEADER]: Becoming Leader. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:10.837016 25020 consensus_queue.cc:237] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:10.838469 24124 leader_election.cc:288] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 4 election: Requested vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:54:10.839599 24027 catalog_manager.cc:5484] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb reported cstate change: term changed from 3 to 4, leader changed from 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) to 227af58cb9554c438ae592baff6af4fb (127.20.65.65). New cstate: current_term: 4 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:11.008781 24984 tablet_copy_client.cc:523] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:11.011729 24984 tablet_bootstrap.cc:492] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367: Bootstrap starting.
I20240502 01:54:11.072273 24984 tablet_bootstrap.cc:492] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=124 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:11.072665 24984 tablet_bootstrap.cc:492] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367: Bootstrap complete.
I20240502 01:54:11.072914 24984 ts_tablet_manager.cc:1367] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367: Time spent bootstrapping tablet: real 0.061s user 0.056s sys 0.004s
I20240502 01:54:11.073508 24984 raft_consensus.cc:348] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } }
I20240502 01:54:11.073693 24984 raft_consensus.cc:729] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 03b70cf3fe3b4a32ad200d00c0c88367, State: Initialized, Role: LEARNER
I20240502 01:54:11.073871 24984 consensus_queue.cc:260] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } }
I20240502 01:54:11.074333 24984 ts_tablet_manager.cc:1398] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:54:11.075098 24217 tablet_copy_service.cc:342] P 9716594f476e436b91b93ceaed4bcf59: Request end of tablet copy session 03b70cf3fe3b4a32ad200d00c0c88367-68d83f47ca7c4a5caeb4ffda5ebc9757 received from {username='slave'} at 127.0.0.1:32800
I20240502 01:54:11.075194 24217 tablet_copy_service.cc:433] P 9716594f476e436b91b93ceaed4bcf59: ending tablet copy session 03b70cf3fe3b4a32ad200d00c0c88367-68d83f47ca7c4a5caeb4ffda5ebc9757 on tablet 68d83f47ca7c4a5caeb4ffda5ebc9757 with peer 03b70cf3fe3b4a32ad200d00c0c88367
I20240502 01:54:11.083058 24989 tablet_copy_client.cc:523] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:11.085501 24989 tablet_bootstrap.cc:492] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477: Bootstrap starting.
I20240502 01:54:11.090049 24995 tablet_copy_client.cc:523] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:11.093190 24995 tablet_bootstrap.cc:492] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565: Bootstrap starting.
I20240502 01:54:11.093806 24993 tablet_copy_client.cc:523] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:11.097380 24993 tablet_bootstrap.cc:492] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477: Bootstrap starting.
I20240502 01:54:11.127635 24989 tablet_bootstrap.cc:492] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=102 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:11.128909 24989 tablet_bootstrap.cc:492] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477: Bootstrap complete.
I20240502 01:54:11.129230 24989 ts_tablet_manager.cc:1367] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477: Time spent bootstrapping tablet: real 0.044s user 0.031s sys 0.005s
I20240502 01:54:11.130051 24989 raft_consensus.cc:348] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:11.130375 24989 raft_consensus.cc:729] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cabbb33d9a564f7ca7c2d5cceeedc477, State: Initialized, Role: LEARNER
I20240502 01:54:11.130609 24989 consensus_queue.cc:260] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:11.132095 24989 ts_tablet_manager.cc:1398] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:11.132759 24291 tablet_copy_service.cc:342] P a6f86494e4de4d3c851c5dab07b40568: Request end of tablet copy session cabbb33d9a564f7ca7c2d5cceeedc477-f0684f060716413796e90d5439d5d482 received from {username='slave'} at 127.0.0.1:58482
I20240502 01:54:11.132864 24291 tablet_copy_service.cc:433] P a6f86494e4de4d3c851c5dab07b40568: ending tablet copy session cabbb33d9a564f7ca7c2d5cceeedc477-f0684f060716413796e90d5439d5d482 on tablet f0684f060716413796e90d5439d5d482 with peer cabbb33d9a564f7ca7c2d5cceeedc477
I20240502 01:54:11.143991 25006 tablet_copy_client.cc:523] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:11.146618 24995 tablet_bootstrap.cc:492] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:11.147071 24995 tablet_bootstrap.cc:492] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565: Bootstrap complete.
I20240502 01:54:11.147302 24995 ts_tablet_manager.cc:1367] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565: Time spent bootstrapping tablet: real 0.054s user 0.030s sys 0.014s
I20240502 01:54:11.147614 25006 tablet_bootstrap.cc:492] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367: Bootstrap starting.
I20240502 01:54:11.147951 24995 raft_consensus.cc:348] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:11.148125 24995 raft_consensus.cc:729] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: a3bcf2c9cfd24f25964eea216f017565, State: Initialized, Role: LEARNER
I20240502 01:54:11.148319 24995 consensus_queue.cc:260] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:11.148763 24995 ts_tablet_manager.cc:1398] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 01:54:11.149302 24217 tablet_copy_service.cc:342] P 9716594f476e436b91b93ceaed4bcf59: Request end of tablet copy session a3bcf2c9cfd24f25964eea216f017565-92d3bd85ba5a4586b9f3ef23824c0287 received from {username='slave'} at 127.0.0.1:32816
I20240502 01:54:11.149400 24217 tablet_copy_service.cc:433] P 9716594f476e436b91b93ceaed4bcf59: ending tablet copy session a3bcf2c9cfd24f25964eea216f017565-92d3bd85ba5a4586b9f3ef23824c0287 on tablet 92d3bd85ba5a4586b9f3ef23824c0287 with peer a3bcf2c9cfd24f25964eea216f017565
I20240502 01:54:11.152387 24993 tablet_bootstrap.cc:492] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=109 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:11.152750 24993 tablet_bootstrap.cc:492] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477: Bootstrap complete.
I20240502 01:54:11.153035 24993 ts_tablet_manager.cc:1367] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477: Time spent bootstrapping tablet: real 0.056s user 0.047s sys 0.002s
I20240502 01:54:11.153770 24993 raft_consensus.cc:348] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:11.154034 24993 raft_consensus.cc:729] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cabbb33d9a564f7ca7c2d5cceeedc477, State: Initialized, Role: LEARNER
I20240502 01:54:11.154260 24993 consensus_queue.cc:260] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:11.154839 24993 ts_tablet_manager.cc:1398] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240502 01:54:11.159199 24692 tablet_copy_service.cc:342] P 03b70cf3fe3b4a32ad200d00c0c88367: Request end of tablet copy session cabbb33d9a564f7ca7c2d5cceeedc477-d516fd48dce443169e7e6011ea6a704d received from {username='slave'} at 127.0.0.1:42956
I20240502 01:54:11.159327 24692 tablet_copy_service.cc:433] P 03b70cf3fe3b4a32ad200d00c0c88367: ending tablet copy session cabbb33d9a564f7ca7c2d5cceeedc477-d516fd48dce443169e7e6011ea6a704d on tablet d516fd48dce443169e7e6011ea6a704d with peer cabbb33d9a564f7ca7c2d5cceeedc477
I20240502 01:54:11.191051 25006 tablet_bootstrap.cc:492] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=97 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:11.191418 25006 tablet_bootstrap.cc:492] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367: Bootstrap complete.
I20240502 01:54:11.191673 25006 ts_tablet_manager.cc:1367] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367: Time spent bootstrapping tablet: real 0.044s user 0.038s sys 0.001s
I20240502 01:54:11.192462 25006 raft_consensus.cc:348] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } }
I20240502 01:54:11.192742 25006 raft_consensus.cc:729] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 03b70cf3fe3b4a32ad200d00c0c88367, State: Initialized, Role: LEARNER
I20240502 01:54:11.192982 25006 consensus_queue.cc:260] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } }
I20240502 01:54:11.193562 25006 ts_tablet_manager.cc:1398] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:11.194237 24618 tablet_copy_service.cc:342] P a3bcf2c9cfd24f25964eea216f017565: Request end of tablet copy session 03b70cf3fe3b4a32ad200d00c0c88367-1edcd38b8d8643ea896111553e49c16c received from {username='slave'} at 127.0.0.1:39962
I20240502 01:54:11.194358 24618 tablet_copy_service.cc:433] P a3bcf2c9cfd24f25964eea216f017565: ending tablet copy session 03b70cf3fe3b4a32ad200d00c0c88367-1edcd38b8d8643ea896111553e49c16c on tablet 1edcd38b8d8643ea896111553e49c16c with peer 03b70cf3fe3b4a32ad200d00c0c88367
I20240502 01:54:11.211947 24681 raft_consensus.cc:1203] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.31->[3.32-3.32] Dedup: 3.32->[]
I20240502 01:54:11.220523 24124 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 5 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240502 01:54:11.221011 25017 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:11.224505 24607 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [term 3 LEARNER]: Advancing to term 5
I20240502 01:54:11.225492 24203 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:11.227452 24607 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [term 5 LEARNER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240502 01:54:11.227954 25017 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:11.228192 24203 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 5 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240502 01:54:11.228727 25017 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:11.246269 25018 raft_consensus.cc:1050] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568: attempting to promote NON_VOTER a3bcf2c9cfd24f25964eea216f017565 to VOTER
I20240502 01:54:11.247092 25018 consensus_queue.cc:237] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 5.31, Last appended by leader: 30, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:11.248203 24532 raft_consensus.cc:1203] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.31->[2.32-2.32] Dedup: 2.32->[]
I20240502 01:54:11.248864 24608 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [term 5 LEARNER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240502 01:54:11.249367 25026 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 01:54:11.250296 24124 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 5 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240502 01:54:11.250828 24203 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 5 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240502 01:54:11.251173 25026 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 01:54:11.251370 25018 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 01:54:11.253553 25017 raft_consensus.cc:2932] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 5 LEADER]: Committing config change with OpId 5.32: config changed from index 29 to 32, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:11.254932 24203 raft_consensus.cc:2932] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 29 to 32, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:11.257970 24124 raft_consensus.cc:2932] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 29 to 32, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:11.259775 24608 raft_consensus.cc:2932] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 29 to 32, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:11.261804 24027 catalog_manager.cc:5484] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: config changed from index 29 to 32, a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:11.267397 24274 consensus_queue.cc:237] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:11.269111 24203 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 5 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240502 01:54:11.269291 24608 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [term 5 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240502 01:54:11.269850 25018 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 01:54:11.270152 25057 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 01:54:11.271943 25018 raft_consensus.cc:2932] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 5 LEADER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 227af58cb9554c438ae592baff6af4fb (127.20.65.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:11.272807 24608 raft_consensus.cc:2932] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 227af58cb9554c438ae592baff6af4fb (127.20.65.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:11.274513 24015 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 92d3bd85ba5a4586b9f3ef23824c0287 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:11.275764 24026 catalog_manager.cc:5484] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: config changed from index 32 to 33, VOTER 227af58cb9554c438ae592baff6af4fb (127.20.65.65) evicted. New cstate: current_term: 5 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:11.275993 24203 raft_consensus.cc:2932] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 227af58cb9554c438ae592baff6af4fb (127.20.65.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } }
I20240502 01:54:11.279719 24120 tablet_service.cc:1514] Processing DeleteTablet for tablet 92d3bd85ba5a4586b9f3ef23824c0287 with delete_type TABLET_DATA_TOMBSTONED (TS 227af58cb9554c438ae592baff6af4fb not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:45056
I20240502 01:54:11.280298 25058 tablet_replica.cc:330] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb: stopping tablet replica
I20240502 01:54:11.280514 25058 raft_consensus.cc:2229] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:11.280802 25058 raft_consensus.cc:2258] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:11.281755 25058 ts_tablet_manager.cc:1875] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:11.286674 25058 ts_tablet_manager.cc:1888] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.32
I20240502 01:54:11.286821 25058 log.cc:1196] T 92d3bd85ba5a4586b9f3ef23824c0287 P 227af58cb9554c438ae592baff6af4fb: Deleting WAL directory at /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-0-root/wals/92d3bd85ba5a4586b9f3ef23824c0287
I20240502 01:54:11.287544 24014 catalog_manager.cc:4830] TS 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551): tablet 92d3bd85ba5a4586b9f3ef23824c0287 (table test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]) successfully deleted
I20240502 01:54:11.289995 24532 raft_consensus.cc:3037] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 LEARNER]: Advancing to term 4
I20240502 01:54:11.292034 24532 raft_consensus.cc:1261] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477 [term 4 LEARNER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240502 01:54:11.292447 25020 consensus_queue.cc:1035] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:54:11.306886 25040 raft_consensus.cc:1050] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb: attempting to promote NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 to VOTER
W20240502 01:54:11.307041 25040 raft_consensus.cc:1054] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb: Unable to promote non-voter cabbb33d9a564f7ca7c2d5cceeedc477: Illegal state: Leader has not yet committed an operation in its own term
I20240502 01:54:11.405315 24203 raft_consensus.cc:1261] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 4 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240502 01:54:11.405864 25040 consensus_queue.cc:1035] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:11.409508 24274 raft_consensus.cc:1261] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 4 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240502 01:54:11.410934 25020 raft_consensus.cc:1050] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb: attempting to promote NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 to VOTER
I20240502 01:54:11.410835 25040 consensus_queue.cc:1035] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:54:11.411623 25020 consensus_queue.cc:237] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 28, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:11.414548 24275 raft_consensus.cc:1261] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 4 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240502 01:54:11.415139 25062 consensus_queue.cc:1035] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:11.415693 24532 raft_consensus.cc:1261] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477 [term 4 LEARNER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240502 01:54:11.416875 24203 raft_consensus.cc:1261] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 4 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240502 01:54:11.416178 25062 consensus_queue.cc:1035] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20240502 01:54:11.417965 25040 consensus_queue.cc:1035] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:11.420166 25020 raft_consensus.cc:2932] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 4 LEADER]: Committing config change with OpId 4.30: config changed from index 27 to 30, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.420796 24532 raft_consensus.cc:2932] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 27 to 30, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.424723 24026 catalog_manager.cc:5484] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477 reported cstate change: config changed from index 27 to 30, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.424578 24203 raft_consensus.cc:2932] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 27 to 30, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.425103 24275 raft_consensus.cc:2932] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 27 to 30, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.432827 24124 consensus_queue.cc:237] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:11.434427 24532 raft_consensus.cc:1261] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477 [term 4 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 01:54:11.434427 24275 raft_consensus.cc:1261] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 4 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 01:54:11.434921 25062 consensus_queue.cc:1035] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:11.435217 25040 consensus_queue.cc:1035] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:11.437547 25020 raft_consensus.cc:2932] T f0684f060716413796e90d5439d5d482 P 227af58cb9554c438ae592baff6af4fb [term 4 LEADER]: Committing config change with OpId 4.31: config changed from index 30 to 31, VOTER 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.438138 24532 raft_consensus.cc:2932] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, VOTER 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.438227 24275 raft_consensus.cc:2932] T f0684f060716413796e90d5439d5d482 P a6f86494e4de4d3c851c5dab07b40568 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, VOTER 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.442255 24027 catalog_manager.cc:5484] T f0684f060716413796e90d5439d5d482 P cabbb33d9a564f7ca7c2d5cceeedc477 reported cstate change: config changed from index 30 to 31, VOTER 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) evicted. New cstate: current_term: 4 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.443859 24014 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f0684f060716413796e90d5439d5d482 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:11.446888 24194 tablet_service.cc:1514] Processing DeleteTablet for tablet f0684f060716413796e90d5439d5d482 with delete_type TABLET_DATA_TOMBSTONED (TS 9716594f476e436b91b93ceaed4bcf59 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:60978
I20240502 01:54:11.447496 25063 tablet_replica.cc:330] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59: stopping tablet replica
I20240502 01:54:11.447710 25063 raft_consensus.cc:2229] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:11.447969 25063 raft_consensus.cc:2258] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:11.448724 25063 ts_tablet_manager.cc:1875] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:11.454025 25063 ts_tablet_manager.cc:1888] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.30
I20240502 01:54:11.454166 25063 log.cc:1196] T f0684f060716413796e90d5439d5d482 P 9716594f476e436b91b93ceaed4bcf59: Deleting WAL directory at /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-1-root/wals/f0684f060716413796e90d5439d5d482
I20240502 01:54:11.454898 24014 catalog_manager.cc:4830] TS 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365): tablet f0684f060716413796e90d5439d5d482 (table test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]) successfully deleted
I20240502 01:54:11.494618 24681 raft_consensus.cc:1203] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.29->[3.30-3.30] Dedup: 3.30->[]
I20240502 01:54:11.686753 25065 raft_consensus.cc:1050] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565: attempting to promote NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 to VOTER
I20240502 01:54:11.687965 25065 consensus_queue.cc:237] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 30, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } }
I20240502 01:54:11.689996 24682 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367 [term 3 LEARNER]: Refusing update from remote peer a3bcf2c9cfd24f25964eea216f017565: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 01:54:11.689996 24203 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Refusing update from remote peer a3bcf2c9cfd24f25964eea216f017565: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 01:54:11.690665 24275 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Refusing update from remote peer a3bcf2c9cfd24f25964eea216f017565: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 01:54:11.690634 25028 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 01:54:11.691164 25028 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 01:54:11.691931 25064 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 01:54:11.705596 25065 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:11.706682 24682 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:11.710762 24022 catalog_manager.cc:5484] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 reported cstate change: config changed from index 32 to 33, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "a3bcf2c9cfd24f25964eea216f017565" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:11.712080 24207 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:11.715934 24608 consensus_queue.cc:237] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } }
I20240502 01:54:11.718041 24207 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Refusing update from remote peer a3bcf2c9cfd24f25964eea216f017565: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240502 01:54:11.706140 24275 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:11.718607 25028 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 01:54:11.721572 25028 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [term 3 LEADER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:11.722174 24207 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:11.726334 24027 catalog_manager.cc:5484] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 reported cstate change: config changed from index 33 to 34, VOTER a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) evicted. New cstate: current_term: 3 leader_uuid: "a3bcf2c9cfd24f25964eea216f017565" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:11.726886 24074 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1edcd38b8d8643ea896111553e49c16c with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:11.730048 24270 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edcd38b8d8643ea896111553e49c16c with delete_type TABLET_DATA_TOMBSTONED (TS a6f86494e4de4d3c851c5dab07b40568 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:55074
I20240502 01:54:11.730693 25070 tablet_replica.cc:330] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568: stopping tablet replica
I20240502 01:54:11.730913 25070 raft_consensus.cc:2229] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:11.731169 25070 raft_consensus.cc:2258] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:11.732491 25070 ts_tablet_manager.cc:1875] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:11.738095 24681 raft_consensus.cc:1261] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367 [term 3 FOLLOWER]: Refusing update from remote peer a3bcf2c9cfd24f25964eea216f017565: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240502 01:54:11.738898 25065 consensus_queue.cc:1035] T 1edcd38b8d8643ea896111553e49c16c P a3bcf2c9cfd24f25964eea216f017565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 01:54:11.740232 24681 raft_consensus.cc:2932] T 1edcd38b8d8643ea896111553e49c16c P 03b70cf3fe3b4a32ad200d00c0c88367 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:11.740649 25070 ts_tablet_manager.cc:1888] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.33
I20240502 01:54:11.740751 25070 log.cc:1196] T 1edcd38b8d8643ea896111553e49c16c P a6f86494e4de4d3c851c5dab07b40568: Deleting WAL directory at /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-2-root/wals/1edcd38b8d8643ea896111553e49c16c
I20240502 01:54:11.741446 24015 catalog_manager.cc:4830] TS a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253): tablet 1edcd38b8d8643ea896111553e49c16c (table test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]) successfully deleted
I20240502 01:54:11.792260 24081 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:54:11.795825 24124 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
mode: GRACEFUL
new_leader_uuid: "9716594f476e436b91b93ceaed4bcf59"
from {username='slave'} at 127.0.0.1:45086
I20240502 01:54:11.796046 24124 raft_consensus.cc:595] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 3 LEADER]: Received request to transfer leadership to TS 9716594f476e436b91b93ceaed4bcf59
I20240502 01:54:11.796991 24275 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287"
dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
mode: GRACEFUL
new_leader_uuid: "a3bcf2c9cfd24f25964eea216f017565"
from {username='slave'} at 127.0.0.1:55106
I20240502 01:54:11.797155 24275 raft_consensus.cc:595] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 5 LEADER]: Received request to transfer leadership to TS a3bcf2c9cfd24f25964eea216f017565
I20240502 01:54:11.798131 24275 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3f5d1021b8b4932b7fe13b7fb933a9c"
dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
mode: GRACEFUL
new_leader_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:55106
I20240502 01:54:11.798290 24275 raft_consensus.cc:595] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Received request to transfer leadership to TS 227af58cb9554c438ae592baff6af4fb
I20240502 01:54:11.798722 24081 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 01:54:11.798938 24081 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:54:11.829368 25061 raft_consensus.cc:979] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb: : Instructing follower 9716594f476e436b91b93ceaed4bcf59 to start an election
I20240502 01:54:11.829552 25061 raft_consensus.cc:1067] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 3 LEADER]: Signalling peer 9716594f476e436b91b93ceaed4bcf59 to start an election
I20240502 01:54:11.830085 24207 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a"
dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
from {username='slave'} at 127.0.0.1:60992
I20240502 01:54:11.830241 24207 raft_consensus.cc:482] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:11.830332 24207 raft_consensus.cc:3037] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:11.832783 24207 raft_consensus.cc:504] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:11.833423 24207 leader_election.cc:288] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 4 election: Requested vote from peers 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551), a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69:38267)
I20240502 01:54:11.833746 24124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "227af58cb9554c438ae592baff6af4fb"
I20240502 01:54:11.833747 24608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf1c78d5f3f406d9029fe29deb6412a" candidate_uuid: "9716594f476e436b91b93ceaed4bcf59" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "a3bcf2c9cfd24f25964eea216f017565"
I20240502 01:54:11.833974 24124 raft_consensus.cc:3032] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:11.834017 24608 raft_consensus.cc:3037] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:11.834082 24124 raft_consensus.cc:729] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:11.834447 24124 consensus_queue.cc:260] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:11.834828 24124 raft_consensus.cc:3037] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:11.836616 24608 raft_consensus.cc:2454] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 4.
I20240502 01:54:11.837056 24171 leader_election.cc:302] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9716594f476e436b91b93ceaed4bcf59, a3bcf2c9cfd24f25964eea216f017565; no voters:
I20240502 01:54:11.837280 24124 raft_consensus.cc:2454] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9716594f476e436b91b93ceaed4bcf59 in term 4.
I20240502 01:54:11.837345 24959 raft_consensus.cc:2781] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:54:11.837745 24959 raft_consensus.cc:686] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 4 LEADER]: Becoming Leader. State: Replica: 9716594f476e436b91b93ceaed4bcf59, State: Running, Role: LEADER
I20240502 01:54:11.837980 24959 consensus_queue.cc:237] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:11.840379 24022 catalog_manager.cc:5484] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: term changed from 3 to 4, leader changed from 227af58cb9554c438ae592baff6af4fb (127.20.65.65) to 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66). New cstate: current_term: 4 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:11.866442 25074 raft_consensus.cc:1050] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367: attempting to promote NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 to VOTER
I20240502 01:54:11.867447 25074 consensus_queue.cc:237] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 30, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:11.869962 24275 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Refusing update from remote peer 03b70cf3fe3b4a32ad200d00c0c88367: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 01:54:11.870522 25076 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 01:54:11.870780 24532 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 LEARNER]: Refusing update from remote peer 03b70cf3fe3b4a32ad200d00c0c88367: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 01:54:11.870785 24207 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Refusing update from remote peer 03b70cf3fe3b4a32ad200d00c0c88367: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240502 01:54:11.871392 25076 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 01:54:11.871752 25074 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 01:54:11.874866 25031 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.876086 24203 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.876166 24532 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.879993 24275 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.880162 24022 catalog_manager.cc:5484] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: config changed from index 32 to 33, cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.887601 24681 consensus_queue.cc:237] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:11.889245 24532 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 FOLLOWER]: Refusing update from remote peer 03b70cf3fe3b4a32ad200d00c0c88367: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240502 01:54:11.889362 24275 raft_consensus.cc:1261] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Refusing update from remote peer 03b70cf3fe3b4a32ad200d00c0c88367: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240502 01:54:11.889883 25076 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 01:54:11.891547 25057 raft_consensus.cc:979] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568: : Instructing follower a3bcf2c9cfd24f25964eea216f017565 to start an election
I20240502 01:54:11.891723 25057 raft_consensus.cc:1067] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 5 LEADER]: Signalling peer a3bcf2c9cfd24f25964eea216f017565 to start an election
I20240502 01:54:11.890334 25074 consensus_queue.cc:1035] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.002s
I20240502 01:54:11.892282 24608 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287"
dest_uuid: "a3bcf2c9cfd24f25964eea216f017565"
from {username='slave'} at 127.0.0.1:54110
I20240502 01:54:11.892454 24608 raft_consensus.cc:482] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:11.892573 24608 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:11.894418 24608 raft_consensus.cc:504] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:11.894981 24608 leader_election.cc:288] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [CANDIDATE]: Term 6 election: Requested vote from peers 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365), a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:54:11.895390 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287" candidate_uuid: "a3bcf2c9cfd24f25964eea216f017565" candidate_term: 6 candidate_status { last_received { term: 5 index: 33 } } ignore_live_leader: true dest_uuid: "9716594f476e436b91b93ceaed4bcf59"
I20240502 01:54:11.895390 24275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d3bd85ba5a4586b9f3ef23824c0287" candidate_uuid: "a3bcf2c9cfd24f25964eea216f017565" candidate_term: 6 candidate_status { last_received { term: 5 index: 33 } } ignore_live_leader: true dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:54:11.895619 24207 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:11.895737 24275 raft_consensus.cc:3032] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 01:54:11.895870 24275 raft_consensus.cc:729] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Running, Role: LEADER
I20240502 01:54:11.896157 24275 consensus_queue.cc:260] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 33, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:11.896458 24275 raft_consensus.cc:3037] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:11.897732 24207 raft_consensus.cc:2454] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3bcf2c9cfd24f25964eea216f017565 in term 6.
I20240502 01:54:11.898172 24572 leader_election.cc:302] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9716594f476e436b91b93ceaed4bcf59, a3bcf2c9cfd24f25964eea216f017565; no voters:
I20240502 01:54:11.898521 25065 raft_consensus.cc:2781] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 01:54:11.898680 24275 raft_consensus.cc:2454] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3bcf2c9cfd24f25964eea216f017565 in term 6.
I20240502 01:54:11.898825 25073 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.899188 25065 raft_consensus.cc:686] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [term 6 LEADER]: Becoming Leader. State: Replica: a3bcf2c9cfd24f25964eea216f017565, State: Running, Role: LEADER
I20240502 01:54:11.899418 25065 consensus_queue.cc:237] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 33, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } }
I20240502 01:54:11.901408 24027 catalog_manager.cc:5484] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 reported cstate change: term changed from 5 to 6, leader changed from a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) to a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69). New cstate: current_term: 6 leader_uuid: "a3bcf2c9cfd24f25964eea216f017565" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:11.902551 24074 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d516fd48dce443169e7e6011ea6a704d with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:11.903172 24022 catalog_manager.cc:5484] T d516fd48dce443169e7e6011ea6a704d P 03b70cf3fe3b4a32ad200d00c0c88367 reported cstate change: config changed from index 33 to 34, VOTER 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) evicted. New cstate: current_term: 2 leader_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:11.904314 24532 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.903892 24275 raft_consensus.cc:2932] T d516fd48dce443169e7e6011ea6a704d P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } }
I20240502 01:54:11.906391 24194 tablet_service.cc:1514] Processing DeleteTablet for tablet d516fd48dce443169e7e6011ea6a704d with delete_type TABLET_DATA_TOMBSTONED (TS 9716594f476e436b91b93ceaed4bcf59 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:60978
I20240502 01:54:11.906733 25063 tablet_replica.cc:330] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59: stopping tablet replica
I20240502 01:54:11.906957 25063 raft_consensus.cc:2229] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:11.907228 25063 raft_consensus.cc:2258] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:11.910063 25063 ts_tablet_manager.cc:1875] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:11.915326 25063 ts_tablet_manager.cc:1888] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20240502 01:54:11.915457 25063 log.cc:1196] T d516fd48dce443169e7e6011ea6a704d P 9716594f476e436b91b93ceaed4bcf59: Deleting WAL directory at /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-1-root/wals/d516fd48dce443169e7e6011ea6a704d
I20240502 01:54:11.916098 24014 catalog_manager.cc:4830] TS 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365): tablet d516fd48dce443169e7e6011ea6a704d (table test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]) successfully deleted
I20240502 01:54:12.091070 25080 raft_consensus.cc:1050] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59: attempting to promote NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 to VOTER
I20240502 01:54:12.091930 25080 consensus_queue.cc:237] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 28, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } }
I20240502 01:54:12.093963 24682 raft_consensus.cc:1261] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 3 LEARNER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 01:54:12.094146 24124 raft_consensus.cc:1261] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 01:54:12.094342 24275 raft_consensus.cc:1261] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 01:54:12.094722 25071 consensus_queue.cc:1035] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:12.095036 25080 consensus_queue.cc:1035] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:12.095345 24959 consensus_queue.cc:1035] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:12.098106 24959 raft_consensus.cc:2932] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 30 to 31, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:12.099524 24682 raft_consensus.cc:2932] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:12.100975 25057 raft_consensus.cc:979] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568: : Instructing follower 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:54:12.101143 25057 raft_consensus.cc:1067] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Signalling peer 227af58cb9554c438ae592baff6af4fb to start an election
I20240502 01:54:12.101716 24124 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3f5d1021b8b4932b7fe13b7fb933a9c"
dest_uuid: "227af58cb9554c438ae592baff6af4fb"
from {username='slave'} at 127.0.0.1:45062
I20240502 01:54:12.101893 24124 raft_consensus.cc:482] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:12.102000 24124 raft_consensus.cc:3037] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:12.103374 24275 raft_consensus.cc:2932] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:12.104264 24027 catalog_manager.cc:5484] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: config changed from index 30 to 31, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:12.104557 24129 raft_consensus.cc:2932] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
W20240502 01:54:12.113703 24959 consensus_queue.cc:1159] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a6f86494e4de4d3c851c5dab07b40568" responder_term: 3 status { last_received { term: 3 index: 31 } last_committed_idx: 31 last_received_current_leader { term: 3 index: 31 } }
I20240502 01:54:12.114285 24207 consensus_queue.cc:237] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } }
I20240502 01:54:12.115062 24124 raft_consensus.cc:504] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:12.115967 24124 leader_election.cc:288] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 3 election: Requested vote from peers a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253), cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68:38415)
I20240502 01:54:12.116186 24275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f5d1021b8b4932b7fe13b7fb933a9c" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "a6f86494e4de4d3c851c5dab07b40568"
I20240502 01:54:12.116501 24532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f5d1021b8b4932b7fe13b7fb933a9c" candidate_uuid: "227af58cb9554c438ae592baff6af4fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477"
I20240502 01:54:12.116688 24532 raft_consensus.cc:3037] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:12.116659 24275 raft_consensus.cc:3032] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:12.116878 24275 raft_consensus.cc:729] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a6f86494e4de4d3c851c5dab07b40568, State: Running, Role: LEADER
I20240502 01:54:12.117194 24275 consensus_queue.cc:260] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:12.117810 24275 raft_consensus.cc:3037] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:12.119293 24532 raft_consensus.cc:2454] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 3.
I20240502 01:54:12.120318 24095 leader_election.cc:302] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227af58cb9554c438ae592baff6af4fb, cabbb33d9a564f7ca7c2d5cceeedc477; no voters:
I20240502 01:54:12.120786 25061 raft_consensus.cc:2781] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:12.120939 25061 raft_consensus.cc:686] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [term 3 LEADER]: Becoming Leader. State: Replica: 227af58cb9554c438ae592baff6af4fb, State: Running, Role: LEADER
I20240502 01:54:12.121212 25061 consensus_queue.cc:237] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } }
I20240502 01:54:12.122822 24682 raft_consensus.cc:1261] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 3 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 01:54:12.123167 24275 raft_consensus.cc:2454] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 227af58cb9554c438ae592baff6af4fb in term 3.
I20240502 01:54:12.123390 25071 consensus_queue.cc:1035] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 01:54:12.124459 24124 raft_consensus.cc:1261] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 01:54:12.126607 25071 consensus_queue.cc:1035] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 01:54:12.127605 24682 raft_consensus.cc:2932] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:12.127977 24022 catalog_manager.cc:5484] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb reported cstate change: term changed from 2 to 3, leader changed from a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) to 227af58cb9554c438ae592baff6af4fb (127.20.65.65). New cstate: current_term: 3 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:12.129611 24124 raft_consensus.cc:2932] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:12.126884 24959 raft_consensus.cc:2932] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 9716594f476e436b91b93ceaed4bcf59 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:12.133811 24022 catalog_manager.cc:5484] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P 227af58cb9554c438ae592baff6af4fb reported cstate change: config changed from index 31 to 32, VOTER a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67) evicted. New cstate: current_term: 3 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } }
I20240502 01:54:12.136184 24014 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 68d83f47ca7c4a5caeb4ffda5ebc9757 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:12.137720 24270 tablet_service.cc:1514] Processing DeleteTablet for tablet 68d83f47ca7c4a5caeb4ffda5ebc9757 with delete_type TABLET_DATA_TOMBSTONED (TS a6f86494e4de4d3c851c5dab07b40568 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:55074
I20240502 01:54:12.138096 25070 tablet_replica.cc:330] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568: stopping tablet replica
I20240502 01:54:12.138342 25070 raft_consensus.cc:2229] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:12.138732 25070 raft_consensus.cc:2258] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:12.139601 25070 ts_tablet_manager.cc:1875] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:12.146366 25070 ts_tablet_manager.cc:1888] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.31
I20240502 01:54:12.146525 25070 log.cc:1196] T 68d83f47ca7c4a5caeb4ffda5ebc9757 P a6f86494e4de4d3c851c5dab07b40568: Deleting WAL directory at /tmp/dist-test-task97s1Zf/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714614741486931-20741-0/minicluster-data/ts-2-root/wals/68d83f47ca7c4a5caeb4ffda5ebc9757
I20240502 01:54:12.147197 24015 catalog_manager.cc:4830] TS a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253): tablet 68d83f47ca7c4a5caeb4ffda5ebc9757 (table test-workload [id=3afd68a8c9de4819b1f3c5ae75cf6307]) successfully deleted
I20240502 01:54:12.318549 24608 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 4 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240502 01:54:12.319129 25086 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 01:54:12.323113 24124 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 4 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240502 01:54:12.323617 24959 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 01:54:12.329020 24275 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P a6f86494e4de4d3c851c5dab07b40568 [term 6 FOLLOWER]: Refusing update from remote peer a3bcf2c9cfd24f25964eea216f017565: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20240502 01:54:12.329449 25064 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 01:54:12.332218 24203 raft_consensus.cc:1261] T 92d3bd85ba5a4586b9f3ef23824c0287 P 9716594f476e436b91b93ceaed4bcf59 [term 6 FOLLOWER]: Refusing update from remote peer a3bcf2c9cfd24f25964eea216f017565: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20240502 01:54:12.332783 25028 consensus_queue.cc:1035] T 92d3bd85ba5a4586b9f3ef23824c0287 P a3bcf2c9cfd24f25964eea216f017565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 01:54:12.724092 24532 raft_consensus.cc:1261] T d3f5d1021b8b4932b7fe13b7fb933a9c P cabbb33d9a564f7ca7c2d5cceeedc477 [term 3 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 01:54:12.724325 24275 raft_consensus.cc:1261] T d3f5d1021b8b4932b7fe13b7fb933a9c P a6f86494e4de4d3c851c5dab07b40568 [term 3 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 01:54:12.724627 25061 consensus_queue.cc:1035] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:12.724901 25095 consensus_queue.cc:1035] T d3f5d1021b8b4932b7fe13b7fb933a9c P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:13.144022 24203 consensus_queue.cc:237] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:13.146188 24275 raft_consensus.cc:1261] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 01:54:13.146201 24608 raft_consensus.cc:1261] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565 [term 2 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
W20240502 01:54:13.146325 24171 consensus_peers.cc:485] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 -> Peer cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68:38415): Couldn't send request to peer cabbb33d9a564f7ca7c2d5cceeedc477. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a64ccf8bf160452c98d7cf42884eadc5. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:13.146754 25086 consensus_queue.cc:1035] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:13.147042 24959 consensus_queue.cc:1035] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:13.149457 25071 raft_consensus.cc:2932] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
I20240502 01:54:13.150053 24608 raft_consensus.cc:2932] T a64ccf8bf160452c98d7cf42884eadc5 P a3bcf2c9cfd24f25964eea216f017565 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
I20240502 01:54:13.150169 24275 raft_consensus.cc:2932] T a64ccf8bf160452c98d7cf42884eadc5 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
I20240502 01:54:13.154635 24022 catalog_manager.cc:5484] T a64ccf8bf160452c98d7cf42884eadc5 P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: config changed from index 29 to 31, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New cstate: current_term: 2 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:13.155912 24281 consensus_queue.cc:237] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:13.158550 24124 raft_consensus.cc:1261] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
W20240502 01:54:13.158565 24246 consensus_peers.cc:485] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 -> Peer a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69:38267): Couldn't send request to peer a3bcf2c9cfd24f25964eea216f017565. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73d4415a5e104edcbfc621ab5604c431. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:13.159214 25026 consensus_queue.cc:1035] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:13.160382 24203 raft_consensus.cc:1261] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 01:54:13.160981 25026 consensus_queue.cc:1035] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:13.165997 25107 raft_consensus.cc:2932] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
I20240502 01:54:13.166767 24203 raft_consensus.cc:2932] T 73d4415a5e104edcbfc621ab5604c431 P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
I20240502 01:54:13.170687 24281 consensus_queue.cc:237] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } }
I20240502 01:54:13.172416 24027 catalog_manager.cc:5484] T 73d4415a5e104edcbfc621ab5604c431 P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: config changed from index -1 to 30, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New cstate: current_term: 3 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:13.175215 24124 raft_consensus.cc:1261] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 01:54:13.175303 24207 raft_consensus.cc:1261] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Refusing update from remote peer a6f86494e4de4d3c851c5dab07b40568: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
W20240502 01:54:13.175666 24246 consensus_peers.cc:485] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 -> Peer 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70:35815): Couldn't send request to peer 03b70cf3fe3b4a32ad200d00c0c88367. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 776becbedf8a4b89a59ea06cf8e2163a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:13.175801 24129 raft_consensus.cc:2932] T 73d4415a5e104edcbfc621ab5604c431 P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
I20240502 01:54:13.176110 25026 consensus_queue.cc:1035] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:13.176602 25026 consensus_queue.cc:1035] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:54:13.179665 25113 raft_consensus.cc:2932] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:54:13.183868 24129 consensus_queue.cc:237] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } }
I20240502 01:54:13.184015 24203 raft_consensus.cc:2932] T 776becbedf8a4b89a59ea06cf8e2163a P 9716594f476e436b91b93ceaed4bcf59 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:54:13.184015 24124 raft_consensus.cc:2932] T 776becbedf8a4b89a59ea06cf8e2163a P 227af58cb9554c438ae592baff6af4fb [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:54:13.185844 24026 catalog_manager.cc:5484] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568 reported cstate change: config changed from index -1 to 30, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New cstate: current_term: 3 leader_uuid: "a6f86494e4de4d3c851c5dab07b40568" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:13.190047 24682 raft_consensus.cc:1261] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 2 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 01:54:13.190593 25095 consensus_queue.cc:1035] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
W20240502 01:54:13.191391 24095 consensus_peers.cc:485] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb -> Peer cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68:38415): Couldn't send request to peer cabbb33d9a564f7ca7c2d5cceeedc477. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc63402f38f94f50b33661f6d776f813. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:13.191382 24281 raft_consensus.cc:1261] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Refusing update from remote peer 227af58cb9554c438ae592baff6af4fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 01:54:13.192046 25095 consensus_queue.cc:1035] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 01:54:13.193534 25100 raft_consensus.cc:2932] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
I20240502 01:54:13.194139 24682 raft_consensus.cc:2932] T dc63402f38f94f50b33661f6d776f813 P 03b70cf3fe3b4a32ad200d00c0c88367 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
I20240502 01:54:13.197629 24027 catalog_manager.cc:5484] T dc63402f38f94f50b33661f6d776f813 P 227af58cb9554c438ae592baff6af4fb reported cstate change: config changed from index 30 to 32, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New cstate: current_term: 2 leader_uuid: "227af58cb9554c438ae592baff6af4fb" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:13.198251 24281 raft_consensus.cc:2932] T dc63402f38f94f50b33661f6d776f813 P a6f86494e4de4d3c851c5dab07b40568 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "a6f86494e4de4d3c851c5dab07b40568" member_type: VOTER last_known_addr { host: "127.20.65.67" port: 33253 } attrs { replace: true } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: NON_VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: true } } }
I20240502 01:54:13.199093 24203 consensus_queue.cc:237] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } }
I20240502 01:54:13.201378 24129 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 4 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240502 01:54:13.201457 24608 raft_consensus.cc:1261] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 4 FOLLOWER]: Refusing update from remote peer 9716594f476e436b91b93ceaed4bcf59: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240502 01:54:13.202090 25071 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
W20240502 01:54:13.202375 24171 consensus_peers.cc:485] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 -> Peer 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70:35815): Couldn't send request to peer 03b70cf3fe3b4a32ad200d00c0c88367. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2cf1c78d5f3f406d9029fe29deb6412a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:13.202570 24959 consensus_queue.cc:1035] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 01:54:13.204460 25086 raft_consensus.cc:2932] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 [term 4 LEADER]: Committing config change with OpId 4.34: config changed from index 30 to 34, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:54:13.205091 24129 raft_consensus.cc:2932] T 2cf1c78d5f3f406d9029fe29deb6412a P 227af58cb9554c438ae592baff6af4fb [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 30 to 34, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:54:13.205258 24608 raft_consensus.cc:2932] T 2cf1c78d5f3f406d9029fe29deb6412a P a3bcf2c9cfd24f25964eea216f017565 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 30 to 34, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } } }
I20240502 01:54:13.210572 24027 catalog_manager.cc:5484] T 2cf1c78d5f3f406d9029fe29deb6412a P 9716594f476e436b91b93ceaed4bcf59 reported cstate change: config changed from index 30 to 34, NON_VOTER 03b70cf3fe3b4a32ad200d00c0c88367 (127.20.65.70) added. New cstate: current_term: 4 leader_uuid: "9716594f476e436b91b93ceaed4bcf59" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03b70cf3fe3b4a32ad200d00c0c88367" member_type: NON_VOTER last_known_addr { host: "127.20.65.70" port: 35815 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:13.220090 24532 consensus_queue.cc:237] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } }
I20240502 01:54:13.228775 24129 raft_consensus.cc:1261] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Refusing update from remote peer cabbb33d9a564f7ca7c2d5cceeedc477: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 01:54:13.228847 24203 raft_consensus.cc:1261] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Refusing update from remote peer cabbb33d9a564f7ca7c2d5cceeedc477: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 01:54:13.229323 25123 consensus_queue.cc:1035] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:13.229656 24927 consensus_queue.cc:1035] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:54:13.232384 25123 raft_consensus.cc:2932] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
I20240502 01:54:13.234705 24129 raft_consensus.cc:2932] T 947791f925a047cf82a9b28983f679bf P 227af58cb9554c438ae592baff6af4fb [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
I20240502 01:54:13.235854 24203 raft_consensus.cc:2932] T 947791f925a047cf82a9b28983f679bf P 9716594f476e436b91b93ceaed4bcf59 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } } }
I20240502 01:54:13.238024 24022 catalog_manager.cc:5484] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 reported cstate change: config changed from index 29 to 31, NON_VOTER a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69) added. New cstate: current_term: 2 leader_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "227af58cb9554c438ae592baff6af4fb" member_type: VOTER last_known_addr { host: "127.20.65.65" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9716594f476e436b91b93ceaed4bcf59" member_type: VOTER last_known_addr { host: "127.20.65.66" port: 37365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cabbb33d9a564f7ca7c2d5cceeedc477" member_type: VOTER last_known_addr { host: "127.20.65.68" port: 38415 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3bcf2c9cfd24f25964eea216f017565" member_type: NON_VOTER last_known_addr { host: "127.20.65.69" port: 38267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:54:13.243041 24497 consensus_peers.cc:485] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477 -> Peer a3bcf2c9cfd24f25964eea216f017565 (127.20.65.69:38267): Couldn't send request to peer a3bcf2c9cfd24f25964eea216f017565. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 947791f925a047cf82a9b28983f679bf. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:13.582594 25129 ts_tablet_manager.cc:902] T 776becbedf8a4b89a59ea06cf8e2163a P 03b70cf3fe3b4a32ad200d00c0c88367: Initiating tablet copy from peer a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
I20240502 01:54:13.583992 25129 tablet_copy_client.cc:308] T 776becbedf8a4b89a59ea06cf8e2163a P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Beginning tablet copy session from remote peer at address 127.20.65.67:33253
I20240502 01:54:13.584563 24291 tablet_copy_service.cc:140] P a6f86494e4de4d3c851c5dab07b40568: Received BeginTabletCopySession request for tablet 776becbedf8a4b89a59ea06cf8e2163a from peer 03b70cf3fe3b4a32ad200d00c0c88367 ({username='slave'} at 127.0.0.1:55122)
I20240502 01:54:13.584726 24291 tablet_copy_service.cc:161] P a6f86494e4de4d3c851c5dab07b40568: Beginning new tablet copy session on tablet 776becbedf8a4b89a59ea06cf8e2163a from peer 03b70cf3fe3b4a32ad200d00c0c88367 at {username='slave'} at 127.0.0.1:55122: session id = 03b70cf3fe3b4a32ad200d00c0c88367-776becbedf8a4b89a59ea06cf8e2163a
I20240502 01:54:13.587436 24291 tablet_copy_source_session.cc:205] T 776becbedf8a4b89a59ea06cf8e2163a P a6f86494e4de4d3c851c5dab07b40568: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:13.588584 25129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776becbedf8a4b89a59ea06cf8e2163a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.593324 25129 tablet_copy_client.cc:783] T 776becbedf8a4b89a59ea06cf8e2163a P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:13.593521 25129 tablet_copy_client.cc:647] T 776becbedf8a4b89a59ea06cf8e2163a P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:13.594914 25133 tablet_copy_client.cc:1002] T 776becbedf8a4b89a59ea06cf8e2163a P 03b70cf3fe3b4a32ad200d00c0c88367: tablet copy: Injecting latency into file download: 2000
I20240502 01:54:13.601876 25134 ts_tablet_manager.cc:902] T 947791f925a047cf82a9b28983f679bf P a3bcf2c9cfd24f25964eea216f017565: Initiating tablet copy from peer cabbb33d9a564f7ca7c2d5cceeedc477 (127.20.65.68:38415)
I20240502 01:54:13.603397 25134 tablet_copy_client.cc:308] T 947791f925a047cf82a9b28983f679bf P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Beginning tablet copy session from remote peer at address 127.20.65.68:38415
I20240502 01:54:13.607908 25141 ts_tablet_manager.cc:902] T 73d4415a5e104edcbfc621ab5604c431 P a3bcf2c9cfd24f25964eea216f017565: Initiating tablet copy from peer a6f86494e4de4d3c851c5dab07b40568 (127.20.65.67:33253)
F20240502 01:54:13.609520 25141 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1edcd38b8d8643ea896111553e49c16c.queue_time_us, tablet-download-pool-1edcd38b8d8643ea896111553e49c16c.queue_time_us) (tablet-download-pool-1edcd38b8d8643ea896111553e49c16c.run_cpu_time_us, tablet-download-pool-1edcd38b8d8643ea896111553e49c16c.run_cpu_time_us) (tablet-download-pool-1edcd38b8d8643ea896111553e49c16c.run_wall_time_us, tablet-download-pool-1edcd38b8d8643ea896111553e49c16c.run_wall_time_us) (tablet-download-pool-2cf1c78d5f3f406d9029fe29deb6412a.queue_time_us, tablet-download-pool-2cf1c78d5f3f406d9029fe29deb6412a.queue_time_us) (tablet-download-pool-2cf1c78d5f3f406d9029fe29deb6412a.run_cpu_time_us, tablet-download-pool-2cf1c78d5f3f406d9029fe29deb6412a.run_cpu_time_us) (tablet-download-pool-2cf1c78d5f3f406d9029fe29deb6412a.run_wall_time_us, tablet-download-pool-2cf1c78d5f3f406d9029fe29deb6412a.run_wall_time_us) (tablet-download-pool-68d83f47ca7c4a5caeb4ffda5ebc9757.queue_time_us, tablet-download-pool-68d83f47ca7c4a5caeb4ffda5ebc9757.queue_time_us) (tablet-download-pool-68d83f47ca7c4a5caeb4ffda5ebc9757.run_cpu_time_us, tablet-download-pool-68d83f47ca7c4a5caeb4ffda5ebc9757.run_cpu_time_us) (tablet-download-pool-68d83f47ca7c4a5caeb4ffda5ebc9757.run_wall_time_us, tablet-download-pool-68d83f47ca7c4a5caeb4ffda5ebc9757.run_wall_time_us) (tablet-download-pool-73d4415a5e104edcbfc621ab5604c431.queue_time_us, tablet-download-pool-73d4415a5e104edcbfc621ab5604c431.queue_time_us) (tablet-download-pool-776becbedf8a4b89a59ea06cf8e2163a.queue_time_us, tablet-download-pool-776becbedf8a4b89a59ea06cf8e2163a.queue_time_us) (tablet-download-pool-776becbedf8a4b89a59ea06cf8e2163a.run_cpu_time_us, tablet-download-pool-776becbedf8a4b89a59ea06cf8e2163a.run_cpu_time_us) (tablet-download-pool-776becbedf8a4b89a59ea06cf8e2163a.run_wall_time_us, tablet-download-pool-776becbedf8a4b89a59ea06cf8e2163a.run_wall_time_us) (tablet-download-pool-7d24cf9c6ad349a3801c6859fa340af1.queue_time_us, tablet-download-pool-7d24cf9c6ad349a3801c6859fa340af1.queue_time_us) (tablet-download-pool-7d24cf9c6ad349a3801c6859fa340af1.run_cpu_time_us, tablet-download-pool-7d24cf9c6ad349a3801c6859fa340af1.run_cpu_time_us) (tablet-download-pool-7d24cf9c6ad349a3801c6859fa340af1.run_wall_time_us, tablet-download-pool-7d24cf9c6ad349a3801c6859fa340af1.run_wall_time_us) (tablet-download-pool-92d3bd85ba5a4586b9f3ef23824c0287.queue_time_us, tablet-download-pool-92d3bd85ba5a4586b9f3ef23824c0287.queue_time_us) (tablet-download-pool-92d3bd85ba5a4586b9f3ef23824c0287.run_cpu_time_us, tablet-download-pool-92d3bd85ba5a4586b9f3ef23824c0287.run_cpu_time_us) (tablet-download-pool-92d3bd85ba5a4586b9f3ef23824c0287.run_wall_time_us, tablet-download-pool-92d3bd85ba5a4586b9f3ef23824c0287.run_wall_time_us) (tablet-download-pool-947791f925a047cf82a9b28983f679bf.queue_time_us, tablet-download-pool-947791f925a047cf82a9b28983f679bf.queue_time_us) (tablet-download-pool-947791f925a047cf82a9b28983f679bf.run_cpu_time_us, tablet-download-pool-947791f925a047cf82a9b28983f679bf.run_cpu_time_us) (tablet-download-pool-947791f925a047cf82a9b28983f679bf.run_wall_time_us, tablet-download-pool-947791f925a047cf82a9b28983f679bf.run_wall_time_us) (tablet-download-pool-a64ccf8bf160452c98d7cf42884eadc5.queue_time_us, tablet-download-pool-a64ccf8bf160452c98d7cf42884eadc5.queue_time_us) (tablet-download-pool-a64ccf8bf160452c98d7cf42884eadc5.run_cpu_time_us, tablet-download-pool-a64ccf8bf160452c98d7cf42884eadc5.run_cpu_time_us) (tablet-download-pool-a64ccf8bf160452c98d7cf42884eadc5.run_wall_time_us, tablet-download-pool-a64ccf8bf160452c98d7cf42884eadc5.run_wall_time_us) (tablet-download-pool-d3f5d1021b8b4932b7fe13b7fb933a9c.queue_time_us, tablet-download-pool-d3f5d1021b8b4932b7fe13b7fb933a9c.queue_time_us) (tablet-download-pool-d3f5d1021b8b4932b7fe13b7fb933a9c.run_cpu_time_us, tablet-download-pool-d3f5d1021b8b4932b7fe13b7fb933a9c.run_cpu_time_us) (tablet-download-pool-d3f5d1021b8b4932b7fe13b7fb933a9c.run_wall_time_us, tablet-download-pool-d3f5d1021b8b4932b7fe13b7fb933a9c.run_wall_time_us) (tablet-download-pool-d516fd48dce443169e7e6011ea6a704d.queue_time_us, tablet-download-pool-d516fd48dce443169e7e6011ea6a704d.queue_time_us) (tablet-download-pool-d516fd48dce443169e7e6011ea6a704d.run_cpu_time_us, tablet-download-pool-d516fd48dce443169e7e6011ea6a704d.run_cpu_time_us) (tablet-download-pool-d516fd48dce443169e7e6011ea6a704d.run_wall_time_us, tablet-download-pool-d516fd48dce443169e7e6011ea6a704d.run_wall_time_us) (tablet-download-pool-dc63402f38f94f50b33661f6d776f813.queue_time_us, tablet-download-pool-dc63402f38f94f50b33661f6d776f813.queue_time_us) (tablet-download-pool-dc63402f38f94f50b33661f6d776f813.run_cpu_time_us, tablet-download-pool-dc63402f38f94f50b33661f6d776f813.run_cpu_time_us) (tablet-download-pool-dc63402f38f94f50b33661f6d776f813.run_wall_time_us, tablet-download-pool-dc63402f38f94f50b33661f6d776f813.run_wall_time_us) (tablet-download-pool-f0684f060716413796e90d5439d5d482.queue_time_us, tablet-download-pool-f0684f060716413796e90d5439d5d482.queue_time_us) (tablet-download-pool-f0684f060716413796e90d5439d5d482.run_cpu_time_us, tablet-download-pool-f0684f060716413796e90d5439d5d482.run_cpu_time_us) (tablet-download-pool-f0684f060716413796e90d5439d5d482.run_wall_time_us, tablet-download-pool-f0684f060716413796e90d5439d5d482.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 01:54:13.609859 24542 tablet_copy_service.cc:140] P cabbb33d9a564f7ca7c2d5cceeedc477: Received BeginTabletCopySession request for tablet 947791f925a047cf82a9b28983f679bf from peer a3bcf2c9cfd24f25964eea216f017565 ({username='slave'} at 127.0.0.1:40186)
*** Aborted at 1714614853 (unix time) try "date -d @1714614853" if you are using GNU date ***
I20240502 01:54:13.611286 24542 tablet_copy_service.cc:161] P cabbb33d9a564f7ca7c2d5cceeedc477: Beginning new tablet copy session on tablet 947791f925a047cf82a9b28983f679bf from peer a3bcf2c9cfd24f25964eea216f017565 at {username='slave'} at 127.0.0.1:40186: session id = a3bcf2c9cfd24f25964eea216f017565-947791f925a047cf82a9b28983f679bf
I20240502 01:54:13.614437 24542 tablet_copy_source_session.cc:205] T 947791f925a047cf82a9b28983f679bf P cabbb33d9a564f7ca7c2d5cceeedc477: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:13.615793 25134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947791f925a047cf82a9b28983f679bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.625617 25134 tablet_copy_client.cc:783] T 947791f925a047cf82a9b28983f679bf P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:13.625862 25134 tablet_copy_client.cc:647] T 947791f925a047cf82a9b28983f679bf P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:13.627504 25139 tablet_copy_client.cc:1002] T 947791f925a047cf82a9b28983f679bf P a3bcf2c9cfd24f25964eea216f017565: tablet copy: Injecting latency into file download: 2000
I20240502 01:54:13.627487 25143 ts_tablet_manager.cc:902] T dc63402f38f94f50b33661f6d776f813 P cabbb33d9a564f7ca7c2d5cceeedc477: Initiating tablet copy from peer 227af58cb9554c438ae592baff6af4fb (127.20.65.65:43551)
I20240502 01:54:13.627683 25142 ts_tablet_manager.cc:902] T 2cf1c78d5f3f406d9029fe29deb6412a P 03b70cf3fe3b4a32ad200d00c0c88367: Initiating tablet copy from peer 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800005105) received by PID 20741 (TID 0x7f7600238700) from PID 20741; stack trace: ***
@ 0x7f76bdf1a980 (unknown) at ??:0
@ 0x7f76b3267fb7 gsignal at ??:0
@ 0x7f76b3269921 abort at ??:0
@ 0x7f76b5d1cdcd google::LogMessage::Fail() at ??:0
@ 0x7f76b5d20b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f76b5d1c7cc google::LogMessage::Flush() at ??:0
@ 0x7f76b5d1df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f76b7399742 kudu::TraceMetrics::InternName() at ??:0
I20240502 01:54:13.669698 25150 ts_tablet_manager.cc:902] T a64ccf8bf160452c98d7cf42884eadc5 P cabbb33d9a564f7ca7c2d5cceeedc477: Initiating tablet copy from peer 9716594f476e436b91b93ceaed4bcf59 (127.20.65.66:37365)
@ 0x7f76b7365f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f76b735fcd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f76c682b8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f76c6843daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f76c696e480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f76c69a6a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f76c6989981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f76c698970e std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f76cb82b98d std::function<>::operator()() at ??:0
@ 0x7f76b7369b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f76b736d19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f76b736d019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f76cb82b98d std::function<>::operator()() at ??:0
@ 0x7f76b734801e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f76bdf0f6db start_thread at ??:0
@ 0x7f76b334a71f clone at ??:0