[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 08:16:35.090051 15684 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.81.62:45287,127.15.81.61:40003,127.15.81.60:38507
I20240502 08:16:35.091830 15684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:35.093001 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:35.104961 15690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.105656 15692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.106312 15689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:35.107134 15684 server_base.cc:1034] running on GCE node
I20240502 08:16:35.107982 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:35.108134 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:35.108202 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637795108186 us; error 0 us; skew 500 ppm
I20240502 08:16:35.108484 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:35.115310 15684 webserver.cc:460] Webserver started at http://127.15.81.62:42765/ using document root <none> and password file <none>
I20240502 08:16:35.116117 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:35.116221 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:35.116554 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:35.119988 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-0-root/instance:
uuid: "75892c56d3fc47989ea5744e676535de"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-rhmj"
I20240502 08:16:35.125708 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240502 08:16:35.129021 15698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.130487 15684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240502 08:16:35.130673 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-0-root
uuid: "75892c56d3fc47989ea5744e676535de"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-rhmj"
I20240502 08:16:35.130868 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:35.146749 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:35.147670 15684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:35.147969 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:35.168191 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.62:45287
I20240502 08:16:35.168213 15749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.62:45287 every 8 connection(s)
I20240502 08:16:35.170483 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:16:35.173594 15750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:35.174010 15752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.174778 15753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.175316 15755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:35.175871 15684 server_base.cc:1034] running on GCE node
I20240502 08:16:35.176219 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:35.176317 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:35.176407 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637795176363 us; error 0 us; skew 500 ppm
I20240502 08:16:35.176614 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:35.178189 15684 webserver.cc:460] Webserver started at http://127.15.81.61:46457/ using document root <none> and password file <none>
I20240502 08:16:35.178450 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:35.178561 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:35.178700 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:35.179517 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-1-root/instance:
uuid: "1753e456a7ff4be4b842b033fbe4607d"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-rhmj"
I20240502 08:16:35.183106 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:35.185371 15760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.185806 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:16:35.185932 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-1-root
uuid: "1753e456a7ff4be4b842b033fbe4607d"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-rhmj"
I20240502 08:16:35.186103 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:35.190268 15750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:35.207667 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:35.208515 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:35.209048 15750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } has no permanent_uuid. Determining permanent_uuid...
W20240502 08:16:35.210394 15701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.81.61:40003: connect: Connection refused (error 111)
W20240502 08:16:35.212173 15750 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.15.81.61:40003: Network error: Client connection negotiation failed: client connection to 127.15.81.61:40003: connect: Connection refused (error 111)
I20240502 08:16:35.226136 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.61:40003
I20240502 08:16:35.226181 15814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.61:40003 every 8 connection(s)
I20240502 08:16:35.227722 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:16:35.227797 15815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:35.231629 15817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:35.231712 15815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } has no permanent_uuid. Determining permanent_uuid...
W20240502 08:16:35.232076 15818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.232677 15820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:35.233378 15684 server_base.cc:1034] running on GCE node
I20240502 08:16:35.233656 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:35.233743 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:35.233834 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637795233783 us; error 0 us; skew 500 ppm
I20240502 08:16:35.234072 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:35.235780 15684 webserver.cc:460] Webserver started at http://127.15.81.60:33063/ using document root <none> and password file <none>
I20240502 08:16:35.236132 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:35.236258 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:35.236433 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:35.237305 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-2-root/instance:
uuid: "49a62bef22ea4b4c958a744d47c0b535"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-rhmj"
I20240502 08:16:35.239828 15815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:35.240868 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:16:35.242902 15827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.243438 15684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:35.243570 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-2-root
uuid: "49a62bef22ea4b4c958a744d47c0b535"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-rhmj"
I20240502 08:16:35.243775 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:35.250056 15815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } has no permanent_uuid. Determining permanent_uuid...
W20240502 08:16:35.252460 15815 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.15.81.60:38507: Network error: Client connection negotiation failed: client connection to 127.15.81.60:38507: connect: Connection refused (error 111)
I20240502 08:16:35.261528 15750 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } attempt: 1
I20240502 08:16:35.267858 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:35.267923 15750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:35.268632 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240502 08:16:35.269963 15750 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.15.81.60:38507: Network error: Client connection negotiation failed: client connection to 127.15.81.60:38507: connect: Connection refused (error 111)
I20240502 08:16:35.284214 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.60:38507
I20240502 08:16:35.284293 15880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.60:38507 every 8 connection(s)
I20240502 08:16:35.285374 15684 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:16:35.285817 15881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.288450 15881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:35.294853 15881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:35.300354 15881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:35.304685 15815 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } attempt: 1
I20240502 08:16:35.312146 15881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535: Bootstrap starting.
I20240502 08:16:35.313191 15750 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } attempt: 1
I20240502 08:16:35.314436 15815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d: Bootstrap starting.
I20240502 08:16:35.317153 15881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.317153 15815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.319098 15881 log.cc:824] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:35.321630 15750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de: Bootstrap starting.
I20240502 08:16:35.322036 15881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535: No bootstrap required, opened a new log
I20240502 08:16:35.322865 15815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d: No bootstrap required, opened a new log
I20240502 08:16:35.323105 15750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.324911 15750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de: No bootstrap required, opened a new log
I20240502 08:16:35.329972 15815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } }
I20240502 08:16:35.329974 15750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } }
I20240502 08:16:35.329974 15881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } }
I20240502 08:16:35.330384 15815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.330405 15750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.330381 15881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.330549 15815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1753e456a7ff4be4b842b033fbe4607d, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.330590 15750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75892c56d3fc47989ea5744e676535de, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.330664 15881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a62bef22ea4b4c958a744d47c0b535, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.331446 15750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } }
I20240502 08:16:35.331476 15815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } }
I20240502 08:16:35.331445 15881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } }
I20240502 08:16:35.332515 15889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } } }
I20240502 08:16:35.332949 15889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:35.333544 15890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } } }
I20240502 08:16:35.333920 15890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:35.334272 15750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:35.334311 15815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:35.334581 15891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } } }
I20240502 08:16:35.334795 15881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:35.335039 15891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [sys.catalog]: This master's current role is: FOLLOWER
W20240502 08:16:35.357725 15908 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 08:16:35.357896 15908 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 08:16:35.360632 15918 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 08:16:35.360774 15918 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 08:16:35.363490 15684 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240502 08:16:35.363658 15684 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240502 08:16:35.364828 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:35.364850 15925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 08:16:35.364988 15925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 08:16:35.369787 15927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.370455 15926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.373735 15930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:35.374568 15684 server_base.cc:1034] running on GCE node
I20240502 08:16:35.374876 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:35.374989 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:35.375052 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637795375030 us; error 0 us; skew 500 ppm
I20240502 08:16:35.375262 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:35.376919 15684 webserver.cc:460] Webserver started at http://127.15.81.1:36623/ using document root <none> and password file <none>
I20240502 08:16:35.377255 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:35.377368 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:35.377543 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:35.378554 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-0-root/instance:
uuid: "2ec0bd3a96f448429fcf481bfc9e2f0c"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-rhmj"
I20240502 08:16:35.382287 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240502 08:16:35.384680 15935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.385125 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:16:35.385258 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-0-root
uuid: "2ec0bd3a96f448429fcf481bfc9e2f0c"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-rhmj"
I20240502 08:16:35.385402 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240502 08:16:35.400224 15704 tablet.cc:2369] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:35.404412 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:35.405086 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:35.406008 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:35.407583 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:35.407750 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.407884 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:35.407979 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.429256 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.1:37819
I20240502 08:16:35.429293 15999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.1:37819 every 8 connection(s)
I20240502 08:16:35.452939 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:16:35.454172 16001 heartbeater.cc:347] Connected to a master server at 127.15.81.62:45287
I20240502 08:16:35.454484 16001 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:35.455881 16001 heartbeater.cc:511] Master 127.15.81.62:45287 requested a full tablet report, sending...
I20240502 08:16:35.457284 16002 heartbeater.cc:347] Connected to a master server at 127.15.81.61:40003
I20240502 08:16:35.457468 16002 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:35.458164 16002 heartbeater.cc:511] Master 127.15.81.61:40003 requested a full tablet report, sending...
I20240502 08:16:35.459004 15715 ts_manager.cc:194] Registered new tserver with Master: 2ec0bd3a96f448429fcf481bfc9e2f0c (127.15.81.1:37819)
I20240502 08:16:35.459378 15780 ts_manager.cc:194] Registered new tserver with Master: 2ec0bd3a96f448429fcf481bfc9e2f0c (127.15.81.1:37819)
W20240502 08:16:35.462965 16007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.464763 16008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:35.465025 16000 heartbeater.cc:347] Connected to a master server at 127.15.81.60:38507
I20240502 08:16:35.465286 16000 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:35.466064 16000 heartbeater.cc:511] Master 127.15.81.60:38507 requested a full tablet report, sending...
I20240502 08:16:35.467710 15846 ts_manager.cc:194] Registered new tserver with Master: 2ec0bd3a96f448429fcf481bfc9e2f0c (127.15.81.1:37819)
I20240502 08:16:35.470633 15684 server_base.cc:1034] running on GCE node
W20240502 08:16:35.471004 16010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:35.471423 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:35.471508 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:35.471601 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637795471557 us; error 0 us; skew 500 ppm
I20240502 08:16:35.471814 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:35.473416 15684 webserver.cc:460] Webserver started at http://127.15.81.2:33567/ using document root <none> and password file <none>
I20240502 08:16:35.473644 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:35.473753 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:35.473899 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:35.474574 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-1-root/instance:
uuid: "1770ebdc5cdb42c0b5a40a24c1ad9fc9"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-rhmj"
I20240502 08:16:35.477411 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:16:35.479676 16015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.480093 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 08:16:35.480218 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-1-root
uuid: "1770ebdc5cdb42c0b5a40a24c1ad9fc9"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-rhmj"
I20240502 08:16:35.480371 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:35.506106 15889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:35.506410 15889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } }
I20240502 08:16:35.508185 15889 leader_election.cc:288] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1753e456a7ff4be4b842b033fbe4607d (127.15.81.61:40003), 49a62bef22ea4b4c958a744d47c0b535 (127.15.81.60:38507)
I20240502 08:16:35.509052 15790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75892c56d3fc47989ea5744e676535de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1753e456a7ff4be4b842b033fbe4607d" is_pre_election: true
I20240502 08:16:35.509115 15856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75892c56d3fc47989ea5744e676535de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49a62bef22ea4b4c958a744d47c0b535" is_pre_election: true
I20240502 08:16:35.509575 15790 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75892c56d3fc47989ea5744e676535de in term 0.
I20240502 08:16:35.509577 15856 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75892c56d3fc47989ea5744e676535de in term 0.
I20240502 08:16:35.510197 15699 leader_election.cc:302] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49a62bef22ea4b4c958a744d47c0b535, 75892c56d3fc47989ea5744e676535de; no voters:
I20240502 08:16:35.510617 15889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:35.510769 15889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:35.510887 15889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.513337 15889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } }
I20240502 08:16:35.513983 15889 leader_election.cc:288] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [CANDIDATE]: Term 1 election: Requested vote from peers 1753e456a7ff4be4b842b033fbe4607d (127.15.81.61:40003), 49a62bef22ea4b4c958a744d47c0b535 (127.15.81.60:38507)
I20240502 08:16:35.513998 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:35.514456 15790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75892c56d3fc47989ea5744e676535de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1753e456a7ff4be4b842b033fbe4607d"
I20240502 08:16:35.514664 15790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.514631 15856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75892c56d3fc47989ea5744e676535de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49a62bef22ea4b4c958a744d47c0b535"
I20240502 08:16:35.514816 15856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.514824 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:35.516286 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:35.518373 15856 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75892c56d3fc47989ea5744e676535de in term 1.
I20240502 08:16:35.518775 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:35.518884 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.518836 15790 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75892c56d3fc47989ea5744e676535de in term 1.
I20240502 08:16:35.518863 15699 leader_election.cc:302] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49a62bef22ea4b4c958a744d47c0b535, 75892c56d3fc47989ea5744e676535de; no voters:
I20240502 08:16:35.519012 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:35.519090 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.519300 15889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:35.520004 15889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [term 1 LEADER]: Becoming Leader. State: Replica: 75892c56d3fc47989ea5744e676535de, State: Running, Role: LEADER
I20240502 08:16:35.520579 15889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } }
I20240502 08:16:35.529570 16024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75892c56d3fc47989ea5744e676535de. Latest consensus state: current_term: 1 leader_uuid: "75892c56d3fc47989ea5744e676535de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } } }
I20240502 08:16:35.529994 16024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:35.534916 16035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:16:35.538420 16035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:16:35.549818 15790 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [term 1 FOLLOWER]: Refusing update from remote peer 75892c56d3fc47989ea5744e676535de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:35.550843 16024 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [LEADER]: Connected to new peer: Peer: permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:35.549829 15856 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [term 1 FOLLOWER]: Refusing update from remote peer 75892c56d3fc47989ea5744e676535de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:35.553745 16024 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [LEADER]: Connected to new peer: Peer: permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:35.561513 15890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75892c56d3fc47989ea5744e676535de. Latest consensus state: current_term: 1 leader_uuid: "75892c56d3fc47989ea5744e676535de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } } }
I20240502 08:16:35.561841 15890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:35.567978 15891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75892c56d3fc47989ea5744e676535de. Latest consensus state: current_term: 1 leader_uuid: "75892c56d3fc47989ea5744e676535de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } } }
I20240502 08:16:35.570188 15891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:35.571530 15889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75892c56d3fc47989ea5744e676535de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } } }
I20240502 08:16:35.571949 15889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:35.572286 15889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75892c56d3fc47989ea5744e676535de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } } }
I20240502 08:16:35.572568 15889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:35.575028 16035 catalog_manager.cc:1347] Generated new cluster ID: c58255c06a584be0b7e36c259154a2e6
I20240502 08:16:35.575176 16035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:16:35.575855 15890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75892c56d3fc47989ea5744e676535de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } } }
I20240502 08:16:35.576159 15890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:35.592114 15891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75892c56d3fc47989ea5744e676535de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75892c56d3fc47989ea5744e676535de" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 45287 } } peers { permanent_uuid: "1753e456a7ff4be4b842b033fbe4607d" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 40003 } } peers { permanent_uuid: "49a62bef22ea4b4c958a744d47c0b535" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 38507 } } }
I20240502 08:16:35.592360 15891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:35.598275 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.2:45533
I20240502 08:16:35.598279 16091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.2:45533 every 8 connection(s)
I20240502 08:16:35.613296 16035 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:16:35.615540 16035 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:16:35.625089 16092 heartbeater.cc:347] Connected to a master server at 127.15.81.60:38507
I20240502 08:16:35.625306 16092 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:35.625962 16092 heartbeater.cc:511] Master 127.15.81.60:38507 requested a full tablet report, sending...
I20240502 08:16:35.627166 15846 ts_manager.cc:194] Registered new tserver with Master: 1770ebdc5cdb42c0b5a40a24c1ad9fc9 (127.15.81.2:45533)
I20240502 08:16:35.633615 16093 heartbeater.cc:347] Connected to a master server at 127.15.81.62:45287
I20240502 08:16:35.633790 16093 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:35.634372 16093 heartbeater.cc:511] Master 127.15.81.62:45287 requested a full tablet report, sending...
I20240502 08:16:35.635476 15715 ts_manager.cc:194] Registered new tserver with Master: 1770ebdc5cdb42c0b5a40a24c1ad9fc9 (127.15.81.2:45533)
I20240502 08:16:35.637492 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:16:35.638927 16095 heartbeater.cc:347] Connected to a master server at 127.15.81.61:40003
I20240502 08:16:35.639120 16095 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:35.639760 16095 heartbeater.cc:511] Master 127.15.81.61:40003 requested a full tablet report, sending...
I20240502 08:16:35.640933 15780 ts_manager.cc:194] Registered new tserver with Master: 1770ebdc5cdb42c0b5a40a24c1ad9fc9 (127.15.81.2:45533)
W20240502 08:16:35.643944 16098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.644758 16099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:35.648635 16035 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de: Generated new TSK 0
I20240502 08:16:35.650130 16035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:16:35.654893 15684 server_base.cc:1034] running on GCE node
W20240502 08:16:35.655421 16101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:35.655870 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:35.655965 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:35.656014 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637795655991 us; error 0 us; skew 500 ppm
I20240502 08:16:35.656189 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:35.657617 15684 webserver.cc:460] Webserver started at http://127.15.81.3:41987/ using document root <none> and password file <none>
I20240502 08:16:35.657868 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:35.657953 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:35.658105 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:35.658747 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-2-root/instance:
uuid: "f485ca230de244e99451a4436575a64b"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-rhmj"
I20240502 08:16:35.661612 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240502 08:16:35.663419 16106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.663813 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 08:16:35.663934 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-2-root
uuid: "f485ca230de244e99451a4436575a64b"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-rhmj"
I20240502 08:16:35.664048 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:35.684420 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:35.685237 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:35.685909 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:35.687016 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:35.687121 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.687239 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:35.687319 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.708972 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.3:34623
I20240502 08:16:35.709026 16169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.3:34623 every 8 connection(s)
I20240502 08:16:35.730635 16171 heartbeater.cc:347] Connected to a master server at 127.15.81.62:45287
I20240502 08:16:35.730835 16171 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:35.731410 16171 heartbeater.cc:511] Master 127.15.81.62:45287 requested a full tablet report, sending...
I20240502 08:16:35.732492 15715 ts_manager.cc:194] Registered new tserver with Master: f485ca230de244e99451a4436575a64b (127.15.81.3:34623)
I20240502 08:16:35.734938 16170 heartbeater.cc:347] Connected to a master server at 127.15.81.60:38507
I20240502 08:16:35.735127 16170 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:35.735051 15715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46350
I20240502 08:16:35.735800 16170 heartbeater.cc:511] Master 127.15.81.60:38507 requested a full tablet report, sending...
I20240502 08:16:35.737030 15846 ts_manager.cc:194] Registered new tserver with Master: f485ca230de244e99451a4436575a64b (127.15.81.3:34623)
I20240502 08:16:35.737771 16172 heartbeater.cc:347] Connected to a master server at 127.15.81.61:40003
I20240502 08:16:35.738032 16172 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:35.738565 16172 heartbeater.cc:511] Master 127.15.81.61:40003 requested a full tablet report, sending...
I20240502 08:16:35.739717 15780 ts_manager.cc:194] Registered new tserver with Master: f485ca230de244e99451a4436575a64b (127.15.81.3:34623)
I20240502 08:16:35.740686 15684 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012617684s
I20240502 08:16:36.359898 15908 catalog_manager.cc:1259] Loaded cluster ID: c58255c06a584be0b7e36c259154a2e6
I20240502 08:16:36.360013 15908 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d: loading cluster ID for follower catalog manager: success
I20240502 08:16:36.362165 15918 catalog_manager.cc:1259] Loaded cluster ID: c58255c06a584be0b7e36c259154a2e6
I20240502 08:16:36.362282 15918 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535: loading cluster ID for follower catalog manager: success
I20240502 08:16:36.363030 15908 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d: acquiring CA information for follower catalog manager: success
I20240502 08:16:36.364737 15908 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 08:16:36.364941 15918 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535: acquiring CA information for follower catalog manager: success
I20240502 08:16:36.365842 15918 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 08:16:36.462781 15715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46326
I20240502 08:16:36.637982 15715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46334
I20240502 08:16:36.737538 16171 heartbeater.cc:503] Master 127.15.81.62:45287 was elected leader, sending a full tablet report...
I20240502 08:16:37.363418 15924 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:16:37.464154 16001 heartbeater.cc:503] Master 127.15.81.62:45287 was elected leader, sending a full tablet report...
I20240502 08:16:37.639272 16093 heartbeater.cc:503] Master 127.15.81.62:45287 was elected leader, sending a full tablet report...
I20240502 08:16:37.796911 15684 test_util.cc:268] Using random seed: 953887752
I20240502 08:16:37.829900 15715 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46364:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:37.866117 16134 tablet_service.cc:1467] Processing CreateTablet for tablet e514264e2afd42b2a9557ee23c8fad50 (DEFAULT_TABLE table=test-workload [id=679b543d012d4eb7946281bd388f288e]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 08:16:37.866421 16048 tablet_service.cc:1467] Processing CreateTablet for tablet 38f7cf4da59e4f22be90cee8efb063a0 (DEFAULT_TABLE table=test-workload [id=679b543d012d4eb7946281bd388f288e]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 08:16:37.867292 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e514264e2afd42b2a9557ee23c8fad50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.867966 16048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f7cf4da59e4f22be90cee8efb063a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.868279 15964 tablet_service.cc:1467] Processing CreateTablet for tablet 62a925c73368430a8834bb4e4a4f65be (DEFAULT_TABLE table=test-workload [id=679b543d012d4eb7946281bd388f288e]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 08:16:37.868772 15964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a925c73368430a8834bb4e4a4f65be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.872202 15963 tablet_service.cc:1467] Processing CreateTablet for tablet eb098499f41a4cec8dfa4d3a8a7bfc7c (DEFAULT_TABLE table=test-workload [id=679b543d012d4eb7946281bd388f288e]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 08:16:37.872886 15963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb098499f41a4cec8dfa4d3a8a7bfc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.880925 16208 tablet_bootstrap.cc:492] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9: Bootstrap starting.
I20240502 08:16:37.882840 16207 tablet_bootstrap.cc:492] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b: Bootstrap starting.
I20240502 08:16:37.884267 16206 tablet_bootstrap.cc:492] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c: Bootstrap starting.
I20240502 08:16:37.885241 16207 tablet_bootstrap.cc:654] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.885275 16208 tablet_bootstrap.cc:654] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.886610 16206 tablet_bootstrap.cc:654] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.888636 16208 tablet_bootstrap.cc:492] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9: No bootstrap required, opened a new log
I20240502 08:16:37.888690 16207 tablet_bootstrap.cc:492] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b: No bootstrap required, opened a new log
I20240502 08:16:37.888931 16208 ts_tablet_manager.cc:1367] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20240502 08:16:37.888936 16207 ts_tablet_manager.cc:1367] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240502 08:16:37.889773 16206 tablet_bootstrap.cc:492] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c: No bootstrap required, opened a new log
I20240502 08:16:37.889802 16208 raft_consensus.cc:348] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1770ebdc5cdb42c0b5a40a24c1ad9fc9" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 45533 } }
I20240502 08:16:37.889935 16206 ts_tablet_manager.cc:1367] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:16:37.889895 16207 raft_consensus.cc:348] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f485ca230de244e99451a4436575a64b" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 34623 } }
I20240502 08:16:37.890026 16208 raft_consensus.cc:374] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.890197 16207 raft_consensus.cc:374] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.890352 16208 raft_consensus.cc:729] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1770ebdc5cdb42c0b5a40a24c1ad9fc9, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.890379 16207 raft_consensus.cc:729] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f485ca230de244e99451a4436575a64b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.890630 16208 consensus_queue.cc:260] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1770ebdc5cdb42c0b5a40a24c1ad9fc9" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 45533 } }
I20240502 08:16:37.890627 16207 consensus_queue.cc:260] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f485ca230de244e99451a4436575a64b" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 34623 } }
I20240502 08:16:37.890874 16208 raft_consensus.cc:388] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:37.890902 16207 raft_consensus.cc:388] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:37.890991 16208 raft_consensus.cc:482] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:37.891016 16207 raft_consensus.cc:482] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:37.890956 16206 raft_consensus.cc:348] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec0bd3a96f448429fcf481bfc9e2f0c" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37819 } }
I20240502 08:16:37.891127 16206 raft_consensus.cc:374] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.891142 16208 raft_consensus.cc:3037] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.891172 16207 raft_consensus.cc:3037] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.891252 16206 raft_consensus.cc:729] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ec0bd3a96f448429fcf481bfc9e2f0c, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.891474 16206 consensus_queue.cc:260] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec0bd3a96f448429fcf481bfc9e2f0c" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37819 } }
I20240502 08:16:37.891726 16206 raft_consensus.cc:388] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:37.891821 16206 raft_consensus.cc:482] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:37.891940 16206 raft_consensus.cc:3037] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.893402 16208 raft_consensus.cc:504] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1770ebdc5cdb42c0b5a40a24c1ad9fc9" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 45533 } }
I20240502 08:16:37.893604 16208 leader_election.cc:302] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1770ebdc5cdb42c0b5a40a24c1ad9fc9; no voters:
I20240502 08:16:37.894145 16208 leader_election.cc:288] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:37.894281 16212 raft_consensus.cc:2781] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:37.894343 16207 raft_consensus.cc:504] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f485ca230de244e99451a4436575a64b" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 34623 } }
I20240502 08:16:37.894587 16207 leader_election.cc:302] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f485ca230de244e99451a4436575a64b; no voters:
I20240502 08:16:37.896633 16206 raft_consensus.cc:504] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec0bd3a96f448429fcf481bfc9e2f0c" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37819 } }
I20240502 08:16:37.896996 16206 leader_election.cc:302] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ec0bd3a96f448429fcf481bfc9e2f0c; no voters:
I20240502 08:16:37.898269 16207 leader_election.cc:288] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:37.898392 16215 raft_consensus.cc:2781] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:37.898424 16212 raft_consensus.cc:686] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9 [term 1 LEADER]: Becoming Leader. State: Replica: 1770ebdc5cdb42c0b5a40a24c1ad9fc9, State: Running, Role: LEADER
I20240502 08:16:37.899855 16212 consensus_queue.cc:237] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1770ebdc5cdb42c0b5a40a24c1ad9fc9" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 45533 } }
I20240502 08:16:37.900666 16206 leader_election.cc:288] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:37.901293 16216 raft_consensus.cc:2781] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:37.904577 16207 ts_tablet_manager.cc:1398] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b: Time spent starting tablet: real 0.015s user 0.007s sys 0.006s
I20240502 08:16:37.905350 16216 raft_consensus.cc:686] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 1 LEADER]: Becoming Leader. State: Replica: 2ec0bd3a96f448429fcf481bfc9e2f0c, State: Running, Role: LEADER
I20240502 08:16:37.905361 16208 ts_tablet_manager.cc:1398] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
I20240502 08:16:37.906181 16215 raft_consensus.cc:686] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b [term 1 LEADER]: Becoming Leader. State: Replica: f485ca230de244e99451a4436575a64b, State: Running, Role: LEADER
I20240502 08:16:37.905769 16216 consensus_queue.cc:237] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec0bd3a96f448429fcf481bfc9e2f0c" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37819 } }
I20240502 08:16:37.906529 16215 consensus_queue.cc:237] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f485ca230de244e99451a4436575a64b" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 34623 } }
I20240502 08:16:37.905818 16206 ts_tablet_manager.cc:1398] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c: Time spent starting tablet: real 0.016s user 0.006s sys 0.004s
I20240502 08:16:37.906989 16206 tablet_bootstrap.cc:492] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c: Bootstrap starting.
I20240502 08:16:37.909502 16206 tablet_bootstrap.cc:654] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.915055 15715 catalog_manager.cc:5484] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1770ebdc5cdb42c0b5a40a24c1ad9fc9 (127.15.81.2). New cstate: current_term: 1 leader_uuid: "1770ebdc5cdb42c0b5a40a24c1ad9fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1770ebdc5cdb42c0b5a40a24c1ad9fc9" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 45533 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:37.922830 15714 catalog_manager.cc:5484] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b reported cstate change: term changed from 0 to 1, leader changed from <none> to f485ca230de244e99451a4436575a64b (127.15.81.3). New cstate: current_term: 1 leader_uuid: "f485ca230de244e99451a4436575a64b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f485ca230de244e99451a4436575a64b" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 34623 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:37.940454 15715 catalog_manager.cc:5484] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ec0bd3a96f448429fcf481bfc9e2f0c (127.15.81.1). New cstate: current_term: 1 leader_uuid: "2ec0bd3a96f448429fcf481bfc9e2f0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec0bd3a96f448429fcf481bfc9e2f0c" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37819 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:37.956144 16206 tablet_bootstrap.cc:492] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c: No bootstrap required, opened a new log
I20240502 08:16:37.956302 16206 ts_tablet_manager.cc:1367] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.007s
I20240502 08:16:37.956910 16206 raft_consensus.cc:348] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec0bd3a96f448429fcf481bfc9e2f0c" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37819 } }
I20240502 08:16:37.957077 16206 raft_consensus.cc:374] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.957131 16206 raft_consensus.cc:729] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ec0bd3a96f448429fcf481bfc9e2f0c, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.957293 16206 consensus_queue.cc:260] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec0bd3a96f448429fcf481bfc9e2f0c" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37819 } }
I20240502 08:16:37.957406 16206 raft_consensus.cc:388] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:37.957490 16206 raft_consensus.cc:482] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:37.957553 16206 raft_consensus.cc:3037] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.959404 16206 raft_consensus.cc:504] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec0bd3a96f448429fcf481bfc9e2f0c" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37819 } }
I20240502 08:16:37.959586 16206 leader_election.cc:302] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ec0bd3a96f448429fcf481bfc9e2f0c; no voters:
I20240502 08:16:37.959821 16206 leader_election.cc:288] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:37.959906 16221 raft_consensus.cc:2781] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:37.960321 16206 ts_tablet_manager.cc:1398] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 08:16:37.960289 16221 raft_consensus.cc:686] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 1 LEADER]: Becoming Leader. State: Replica: 2ec0bd3a96f448429fcf481bfc9e2f0c, State: Running, Role: LEADER
I20240502 08:16:37.960673 16221 consensus_queue.cc:237] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec0bd3a96f448429fcf481bfc9e2f0c" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37819 } }
I20240502 08:16:37.963079 15715 catalog_manager.cc:5484] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ec0bd3a96f448429fcf481bfc9e2f0c (127.15.81.1). New cstate: current_term: 1 leader_uuid: "2ec0bd3a96f448429fcf481bfc9e2f0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec0bd3a96f448429fcf481bfc9e2f0c" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37819 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:37.978281 15684 tablet_server.cc:178] TabletServer@127.15.81.1:0 shutting down...
I20240502 08:16:37.989485 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:16:37.989939 15684 tablet_replica.cc:330] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c: stopping tablet replica
I20240502 08:16:37.990350 15684 raft_consensus.cc:2229] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:37.990598 15684 raft_consensus.cc:2258] T eb098499f41a4cec8dfa4d3a8a7bfc7c P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.992156 15684 tablet_replica.cc:330] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c: stopping tablet replica
I20240502 08:16:37.992293 15684 raft_consensus.cc:2229] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:37.992429 15684 raft_consensus.cc:2258] T 62a925c73368430a8834bb4e4a4f65be P 2ec0bd3a96f448429fcf481bfc9e2f0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.008333 15684 tablet_server.cc:194] TabletServer@127.15.81.1:0 shutdown complete.
I20240502 08:16:38.014099 15684 tablet_server.cc:178] TabletServer@127.15.81.2:0 shutting down...
I20240502 08:16:38.024339 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:16:38.024791 15684 tablet_replica.cc:330] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9: stopping tablet replica
I20240502 08:16:38.024960 15684 raft_consensus.cc:2229] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:38.025138 15684 raft_consensus.cc:2258] T 38f7cf4da59e4f22be90cee8efb063a0 P 1770ebdc5cdb42c0b5a40a24c1ad9fc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.039966 15684 tablet_server.cc:194] TabletServer@127.15.81.2:0 shutdown complete.
I20240502 08:16:38.043213 15684 tablet_server.cc:178] TabletServer@127.15.81.3:0 shutting down...
I20240502 08:16:38.052250 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:16:38.052677 15684 tablet_replica.cc:330] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b: stopping tablet replica
I20240502 08:16:38.052913 15684 raft_consensus.cc:2229] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:38.053121 15684 raft_consensus.cc:2258] T e514264e2afd42b2a9557ee23c8fad50 P f485ca230de244e99451a4436575a64b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.067481 15684 tablet_server.cc:194] TabletServer@127.15.81.3:0 shutdown complete.
I20240502 08:16:38.070574 15684 master.cc:537] Master@127.15.81.62:45287 shutting down...
I20240502 08:16:38.078907 15684 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:38.079401 15684 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.079569 15684 tablet_replica.cc:330] T 00000000000000000000000000000000 P 75892c56d3fc47989ea5744e676535de: stopping tablet replica
I20240502 08:16:38.095145 15684 master.cc:552] Master@127.15.81.62:45287 shutdown complete.
I20240502 08:16:38.100840 15684 master.cc:537] Master@127.15.81.61:40003 shutting down...
I20240502 08:16:38.108595 15684 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.108858 15684 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.108978 15684 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1753e456a7ff4be4b842b033fbe4607d: stopping tablet replica
I20240502 08:16:38.123085 15684 master.cc:552] Master@127.15.81.61:40003 shutdown complete.
I20240502 08:16:38.127393 15684 master.cc:537] Master@127.15.81.60:38507 shutting down...
I20240502 08:16:38.134001 15684 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.134228 15684 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.134356 15684 tablet_replica.cc:330] T 00000000000000000000000000000000 P 49a62bef22ea4b4c958a744d47c0b535: stopping tablet replica
I20240502 08:16:38.148273 15684 master.cc:552] Master@127.15.81.60:38507 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (3074 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240502 08:16:38.162474 15684 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.81.62:44149
I20240502 08:16:38.163007 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:38.165995 16230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:38.166607 16231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:38.167313 15684 server_base.cc:1034] running on GCE node
W20240502 08:16:38.168154 16233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:38.168524 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:38.168597 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:38.168658 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637798168639 us; error 0 us; skew 500 ppm
I20240502 08:16:38.168860 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:38.170176 15684 webserver.cc:460] Webserver started at http://127.15.81.62:45339/ using document root <none> and password file <none>
I20240502 08:16:38.170428 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:38.170506 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:38.170667 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:38.171344 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/master-0-root/instance:
uuid: "cdf6995650b043faa686c00a8280fc8d"
format_stamp: "Formatted at 2024-05-02 08:16:38 on dist-test-slave-rhmj"
I20240502 08:16:38.173871 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:38.175499 16238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:38.176069 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:16:38.176200 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/master-0-root
uuid: "cdf6995650b043faa686c00a8280fc8d"
format_stamp: "Formatted at 2024-05-02 08:16:38 on dist-test-slave-rhmj"
I20240502 08:16:38.176316 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:38.187333 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:38.188011 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:38.202819 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.62:44149
I20240502 08:16:38.202852 16290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.62:44149 every 8 connection(s)
I20240502 08:16:38.204236 16291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.208434 16291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d: Bootstrap starting.
I20240502 08:16:38.209736 16291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.211318 16291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d: No bootstrap required, opened a new log
I20240502 08:16:38.211946 16291 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf6995650b043faa686c00a8280fc8d" member_type: VOTER }
I20240502 08:16:38.212083 16291 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.212160 16291 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdf6995650b043faa686c00a8280fc8d, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.212364 16291 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf6995650b043faa686c00a8280fc8d" member_type: VOTER }
I20240502 08:16:38.212513 16291 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:38.212574 16291 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:38.212653 16291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:38.214596 16291 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf6995650b043faa686c00a8280fc8d" member_type: VOTER }
I20240502 08:16:38.214828 16291 leader_election.cc:302] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdf6995650b043faa686c00a8280fc8d; no voters:
I20240502 08:16:38.215214 16291 leader_election.cc:288] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:38.215356 16294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:38.215895 16294 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [term 1 LEADER]: Becoming Leader. State: Replica: cdf6995650b043faa686c00a8280fc8d, State: Running, Role: LEADER
I20240502 08:16:38.216173 16294 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf6995650b043faa686c00a8280fc8d" member_type: VOTER }
I20240502 08:16:38.216236 16291 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:38.217144 16295 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdf6995650b043faa686c00a8280fc8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf6995650b043faa686c00a8280fc8d" member_type: VOTER } }
I20240502 08:16:38.217222 16296 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdf6995650b043faa686c00a8280fc8d. Latest consensus state: current_term: 1 leader_uuid: "cdf6995650b043faa686c00a8280fc8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf6995650b043faa686c00a8280fc8d" member_type: VOTER } }
I20240502 08:16:38.217485 16295 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:38.217608 16296 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:38.218302 16300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:16:38.221118 16300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:16:38.222739 15684 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:16:38.224716 16300 catalog_manager.cc:1347] Generated new cluster ID: f794c4de709747adb4ee05d9ae1f2312
I20240502 08:16:38.224792 16300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:16:38.242437 16300 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:16:38.243184 16300 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:16:38.253355 16300 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d: Generated new TSK 0
I20240502 08:16:38.253577 16300 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:16:38.255616 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:38.259109 16313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:38.259475 16314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:38.260200 16316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:38.260738 15684 server_base.cc:1034] running on GCE node
I20240502 08:16:38.260995 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:38.261080 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:38.261159 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637798261124 us; error 0 us; skew 500 ppm
I20240502 08:16:38.261327 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:38.262707 15684 webserver.cc:460] Webserver started at http://127.15.81.1:37015/ using document root <none> and password file <none>
I20240502 08:16:38.262950 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:38.263052 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:38.263212 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:38.263959 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-0-root/instance:
uuid: "f78a635b286d46618742689bcf176adb"
format_stamp: "Formatted at 2024-05-02 08:16:38 on dist-test-slave-rhmj"
I20240502 08:16:38.266582 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:16:38.268265 16321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:38.268685 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240502 08:16:38.268785 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-0-root
uuid: "f78a635b286d46618742689bcf176adb"
format_stamp: "Formatted at 2024-05-02 08:16:38 on dist-test-slave-rhmj"
I20240502 08:16:38.268909 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:38.295464 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:38.296171 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:38.296779 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:38.297789 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:38.297855 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:38.297955 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:38.298017 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:38.314312 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.1:44913
I20240502 08:16:38.314332 16384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.1:44913 every 8 connection(s)
I20240502 08:16:38.316298 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:38.320554 16389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:38.321117 16390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:38.321779 16392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:38.322430 15684 server_base.cc:1034] running on GCE node
I20240502 08:16:38.323148 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:38.323256 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:38.323325 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637798323297 us; error 0 us; skew 500 ppm
I20240502 08:16:38.323509 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:38.324576 16385 heartbeater.cc:347] Connected to a master server at 127.15.81.62:44149
I20240502 08:16:38.324718 16385 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:38.325170 15684 webserver.cc:460] Webserver started at http://127.15.81.2:38427/ using document root <none> and password file <none>
I20240502 08:16:38.325232 16385 heartbeater.cc:511] Master 127.15.81.62:44149 requested a full tablet report, sending...
I20240502 08:16:38.325448 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:38.325538 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:38.325722 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:38.326234 16255 ts_manager.cc:194] Registered new tserver with Master: f78a635b286d46618742689bcf176adb (127.15.81.1:44913)
I20240502 08:16:38.326455 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-1-root/instance:
uuid: "e57450ee55d64a8a87833c73acaef178"
format_stamp: "Formatted at 2024-05-02 08:16:38 on dist-test-slave-rhmj"
I20240502 08:16:38.327414 16255 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38312
I20240502 08:16:38.329236 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 08:16:38.330806 16398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:38.331141 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:16:38.331275 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-1-root
uuid: "e57450ee55d64a8a87833c73acaef178"
format_stamp: "Formatted at 2024-05-02 08:16:38 on dist-test-slave-rhmj"
I20240502 08:16:38.331396 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:38.342425 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:38.343050 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:38.343675 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:38.344545 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:38.344638 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:38.344745 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:38.344825 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:38.358954 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.2:43791
I20240502 08:16:38.359004 16461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.2:43791 every 8 connection(s)
I20240502 08:16:38.360690 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:38.364851 16465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:38.365485 16466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:38.366191 16468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:38.366756 15684 server_base.cc:1034] running on GCE node
I20240502 08:16:38.367081 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:38.367166 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:38.367254 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637798367208 us; error 0 us; skew 500 ppm
I20240502 08:16:38.367425 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:38.368844 16462 heartbeater.cc:347] Connected to a master server at 127.15.81.62:44149
I20240502 08:16:38.368968 16462 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:38.369410 16462 heartbeater.cc:511] Master 127.15.81.62:44149 requested a full tablet report, sending...
I20240502 08:16:38.369837 15684 webserver.cc:460] Webserver started at http://127.15.81.3:46093/ using document root <none> and password file <none>
I20240502 08:16:38.370158 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:38.370307 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:38.370378 16255 ts_manager.cc:194] Registered new tserver with Master: e57450ee55d64a8a87833c73acaef178 (127.15.81.2:43791)
I20240502 08:16:38.370471 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:38.371356 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-2-root/instance:
uuid: "1fc4bb2518f84cedb425773582dbe557"
format_stamp: "Formatted at 2024-05-02 08:16:38 on dist-test-slave-rhmj"
I20240502 08:16:38.371407 16255 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38324
I20240502 08:16:38.374047 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:38.375645 16473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:38.376068 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 08:16:38.376181 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-2-root
uuid: "1fc4bb2518f84cedb425773582dbe557"
format_stamp: "Formatted at 2024-05-02 08:16:38 on dist-test-slave-rhmj"
I20240502 08:16:38.376335 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:38.394301 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:38.394977 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:38.395589 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:38.396595 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:38.396673 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:38.396800 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:38.396867 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:38.410936 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.3:43159
I20240502 08:16:38.410971 16535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.3:43159 every 8 connection(s)
I20240502 08:16:38.418273 16536 heartbeater.cc:347] Connected to a master server at 127.15.81.62:44149
I20240502 08:16:38.418428 16536 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:38.418946 16536 heartbeater.cc:511] Master 127.15.81.62:44149 requested a full tablet report, sending...
I20240502 08:16:38.419912 16255 ts_manager.cc:194] Registered new tserver with Master: 1fc4bb2518f84cedb425773582dbe557 (127.15.81.3:43159)
I20240502 08:16:38.420676 16255 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38338
I20240502 08:16:38.420903 15684 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.009025307s
I20240502 08:16:39.328727 16385 heartbeater.cc:503] Master 127.15.81.62:44149 was elected leader, sending a full tablet report...
I20240502 08:16:39.372843 16462 heartbeater.cc:503] Master 127.15.81.62:44149 was elected leader, sending a full tablet report...
I20240502 08:16:39.422123 16536 heartbeater.cc:503] Master 127.15.81.62:44149 was elected leader, sending a full tablet report...
I20240502 08:16:39.449404 15684 test_util.cc:268] Using random seed: 955540258
I20240502 08:16:39.462320 16255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50378:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:39.463153 16255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.489936 16425 tablet_service.cc:1467] Processing CreateTablet for tablet 496e5ac0593c4ea785c3ad477076b18a (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 08:16:39.490270 16426 tablet_service.cc:1467] Processing CreateTablet for tablet e5a24bcd27534faaaa59a37883434723 (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 08:16:39.490585 16425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 496e5ac0593c4ea785c3ad477076b18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.492061 16426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a24bcd27534faaaa59a37883434723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.492641 16421 tablet_service.cc:1467] Processing CreateTablet for tablet c4003407906b496b89662cfaf89d66be (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 08:16:39.493121 16349 tablet_service.cc:1467] Processing CreateTablet for tablet e5a24bcd27534faaaa59a37883434723 (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 08:16:39.493491 16422 tablet_service.cc:1467] Processing CreateTablet for tablet 96eecf3a17634ff39abdce6c99f4eaa6 (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 08:16:39.494037 16423 tablet_service.cc:1467] Processing CreateTablet for tablet 020165b5b7c743039aa2b3a5a818d17b (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 08:16:39.494904 16423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020165b5b7c743039aa2b3a5a818d17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.496528 16420 tablet_service.cc:1467] Processing CreateTablet for tablet e5a2f2b6053d410b9a51b54869c45901 (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 08:16:39.497159 16349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a24bcd27534faaaa59a37883434723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.496054 16424 tablet_service.cc:1467] Processing CreateTablet for tablet 0b29b3bd78b247c09112e090e1144d52 (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 08:16:39.497326 16348 tablet_service.cc:1467] Processing CreateTablet for tablet 496e5ac0593c4ea785c3ad477076b18a (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 08:16:39.497694 16419 tablet_service.cc:1467] Processing CreateTablet for tablet b6db3ef6e54444309c7e3684428598a0 (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 08:16:39.498515 16421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4003407906b496b89662cfaf89d66be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.498214 16345 tablet_service.cc:1467] Processing CreateTablet for tablet 96eecf3a17634ff39abdce6c99f4eaa6 (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 08:16:39.500213 16345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96eecf3a17634ff39abdce6c99f4eaa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.500161 16346 tablet_service.cc:1467] Processing CreateTablet for tablet 020165b5b7c743039aa2b3a5a818d17b (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 08:16:39.501787 16420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a2f2b6053d410b9a51b54869c45901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.502156 16347 tablet_service.cc:1467] Processing CreateTablet for tablet 0b29b3bd78b247c09112e090e1144d52 (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 08:16:39.502183 16348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 496e5ac0593c4ea785c3ad477076b18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.503521 16499 tablet_service.cc:1467] Processing CreateTablet for tablet 0b29b3bd78b247c09112e090e1144d52 (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 08:16:39.503734 16501 tablet_service.cc:1467] Processing CreateTablet for tablet e5a24bcd27534faaaa59a37883434723 (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 08:16:39.505015 16497 tablet_service.cc:1467] Processing CreateTablet for tablet 96eecf3a17634ff39abdce6c99f4eaa6 (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 08:16:39.505599 16497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96eecf3a17634ff39abdce6c99f4eaa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.506152 16347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b29b3bd78b247c09112e090e1144d52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.506179 16419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6db3ef6e54444309c7e3684428598a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.506825 16422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96eecf3a17634ff39abdce6c99f4eaa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.507634 16496 tablet_service.cc:1467] Processing CreateTablet for tablet c4003407906b496b89662cfaf89d66be (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 08:16:39.508469 16494 tablet_service.cc:1467] Processing CreateTablet for tablet b6db3ef6e54444309c7e3684428598a0 (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 08:16:39.508754 16343 tablet_service.cc:1467] Processing CreateTablet for tablet e5a2f2b6053d410b9a51b54869c45901 (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 08:16:39.508816 16495 tablet_service.cc:1467] Processing CreateTablet for tablet e5a2f2b6053d410b9a51b54869c45901 (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 08:16:39.509078 16344 tablet_service.cc:1467] Processing CreateTablet for tablet c4003407906b496b89662cfaf89d66be (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 08:16:39.509574 16498 tablet_service.cc:1467] Processing CreateTablet for tablet 020165b5b7c743039aa2b3a5a818d17b (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 08:16:39.508858 16500 tablet_service.cc:1467] Processing CreateTablet for tablet 496e5ac0593c4ea785c3ad477076b18a (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 08:16:39.514560 16499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b29b3bd78b247c09112e090e1144d52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.514628 16342 tablet_service.cc:1467] Processing CreateTablet for tablet b6db3ef6e54444309c7e3684428598a0 (DEFAULT_TABLE table=test-workload [id=6562567644fc420fb9dac3d8db9be8c7]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 08:16:39.515117 16346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020165b5b7c743039aa2b3a5a818d17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.516323 16424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b29b3bd78b247c09112e090e1144d52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.516774 16343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a2f2b6053d410b9a51b54869c45901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.518594 16501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a24bcd27534faaaa59a37883434723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.531128 16344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4003407906b496b89662cfaf89d66be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.531240 16496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4003407906b496b89662cfaf89d66be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.532425 16342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6db3ef6e54444309c7e3684428598a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.535073 16500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 496e5ac0593c4ea785c3ad477076b18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.536046 16498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020165b5b7c743039aa2b3a5a818d17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.536419 16495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a2f2b6053d410b9a51b54869c45901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.536717 16494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6db3ef6e54444309c7e3684428598a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.556797 16558 tablet_bootstrap.cc:492] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb: Bootstrap starting.
I20240502 08:16:39.560616 16558 tablet_bootstrap.cc:654] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.570595 16556 tablet_bootstrap.cc:492] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178: Bootstrap starting.
I20240502 08:16:39.577131 16557 tablet_bootstrap.cc:492] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557: Bootstrap starting.
I20240502 08:16:39.584632 16556 tablet_bootstrap.cc:654] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.596107 16556 tablet_bootstrap.cc:492] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178: No bootstrap required, opened a new log
I20240502 08:16:39.596899 16556 ts_tablet_manager.cc:1367] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
I20240502 08:16:39.598974 16557 tablet_bootstrap.cc:654] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.603302 16558 tablet_bootstrap.cc:492] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb: No bootstrap required, opened a new log
I20240502 08:16:39.607828 16558 ts_tablet_manager.cc:1367] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb: Time spent bootstrapping tablet: real 0.047s user 0.009s sys 0.009s
I20240502 08:16:39.607873 16556 raft_consensus.cc:348] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.608162 16556 raft_consensus.cc:374] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.608275 16556 raft_consensus.cc:729] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57450ee55d64a8a87833c73acaef178, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.608556 16556 consensus_queue.cc:260] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.608834 16557 tablet_bootstrap.cc:492] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557: No bootstrap required, opened a new log
I20240502 08:16:39.608780 16558 raft_consensus.cc:348] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.609019 16557 ts_tablet_manager.cc:1367] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.005s
I20240502 08:16:39.609030 16558 raft_consensus.cc:374] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.609280 16558 raft_consensus.cc:729] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78a635b286d46618742689bcf176adb, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.609586 16556 ts_tablet_manager.cc:1398] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240502 08:16:39.609604 16558 consensus_queue.cc:260] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.609900 16556 tablet_bootstrap.cc:492] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178: Bootstrap starting.
I20240502 08:16:39.609970 16557 raft_consensus.cc:348] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.610159 16557 raft_consensus.cc:374] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.610276 16557 raft_consensus.cc:729] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc4bb2518f84cedb425773582dbe557, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.610550 16557 consensus_queue.cc:260] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.612275 16556 tablet_bootstrap.cc:654] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.617018 16558 ts_tablet_manager.cc:1398] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb: Time spent starting tablet: real 0.009s user 0.001s sys 0.006s
I20240502 08:16:39.617453 16558 tablet_bootstrap.cc:492] T e5a24bcd27534faaaa59a37883434723 P f78a635b286d46618742689bcf176adb: Bootstrap starting.
I20240502 08:16:39.617579 16557 ts_tablet_manager.cc:1398] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557: Time spent starting tablet: real 0.008s user 0.001s sys 0.007s
I20240502 08:16:39.618570 16557 tablet_bootstrap.cc:492] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557: Bootstrap starting.
I20240502 08:16:39.619899 16558 tablet_bootstrap.cc:654] T e5a24bcd27534faaaa59a37883434723 P f78a635b286d46618742689bcf176adb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.620502 16557 tablet_bootstrap.cc:654] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.622948 16556 tablet_bootstrap.cc:492] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178: No bootstrap required, opened a new log
I20240502 08:16:39.623148 16556 ts_tablet_manager.cc:1367] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20240502 08:16:39.623337 16557 tablet_bootstrap.cc:492] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557: No bootstrap required, opened a new log
I20240502 08:16:39.623503 16557 ts_tablet_manager.cc:1367] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:16:39.624049 16556 raft_consensus.cc:348] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.624181 16558 tablet_bootstrap.cc:492] T e5a24bcd27534faaaa59a37883434723 P f78a635b286d46618742689bcf176adb: No bootstrap required, opened a new log
I20240502 08:16:39.624300 16556 raft_consensus.cc:374] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.624382 16558 ts_tablet_manager.cc:1367] T e5a24bcd27534faaaa59a37883434723 P f78a635b286d46618742689bcf176adb: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240502 08:16:39.624403 16556 raft_consensus.cc:729] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57450ee55d64a8a87833c73acaef178, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.624308 16557 raft_consensus.cc:348] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.624600 16557 raft_consensus.cc:374] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.624689 16557 raft_consensus.cc:729] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc4bb2518f84cedb425773582dbe557, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.624742 16556 consensus_queue.cc:260] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.624950 16557 consensus_queue.cc:260] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.625156 16558 raft_consensus.cc:348] T e5a24bcd27534faaaa59a37883434723 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } }
I20240502 08:16:39.625366 16556 ts_tablet_manager.cc:1398] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:39.625519 16557 ts_tablet_manager.cc:1398] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:39.625675 16556 tablet_bootstrap.cc:492] T 496e5ac0593c4ea785c3ad477076b18a P e57450ee55d64a8a87833c73acaef178: Bootstrap starting.
I20240502 08:16:39.625788 16557 tablet_bootstrap.cc:492] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557: Bootstrap starting.
I20240502 08:16:39.625388 16558 raft_consensus.cc:374] T e5a24bcd27534faaaa59a37883434723 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.626057 16558 raft_consensus.cc:729] T e5a24bcd27534faaaa59a37883434723 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78a635b286d46618742689bcf176adb, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.626278 16558 consensus_queue.cc:260] T e5a24bcd27534faaaa59a37883434723 P f78a635b286d46618742689bcf176adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } }
I20240502 08:16:39.627499 16556 tablet_bootstrap.cc:654] T 496e5ac0593c4ea785c3ad477076b18a P e57450ee55d64a8a87833c73acaef178: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.627513 16557 tablet_bootstrap.cc:654] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.627722 16558 ts_tablet_manager.cc:1398] T e5a24bcd27534faaaa59a37883434723 P f78a635b286d46618742689bcf176adb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:39.628065 16558 tablet_bootstrap.cc:492] T e5a2f2b6053d410b9a51b54869c45901 P f78a635b286d46618742689bcf176adb: Bootstrap starting.
I20240502 08:16:39.630055 16558 tablet_bootstrap.cc:654] T e5a2f2b6053d410b9a51b54869c45901 P f78a635b286d46618742689bcf176adb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.638736 16558 tablet_bootstrap.cc:492] T e5a2f2b6053d410b9a51b54869c45901 P f78a635b286d46618742689bcf176adb: No bootstrap required, opened a new log
I20240502 08:16:39.638984 16558 ts_tablet_manager.cc:1367] T e5a2f2b6053d410b9a51b54869c45901 P f78a635b286d46618742689bcf176adb: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240502 08:16:39.639503 16556 tablet_bootstrap.cc:492] T 496e5ac0593c4ea785c3ad477076b18a P e57450ee55d64a8a87833c73acaef178: No bootstrap required, opened a new log
I20240502 08:16:39.639700 16556 ts_tablet_manager.cc:1367] T 496e5ac0593c4ea785c3ad477076b18a P e57450ee55d64a8a87833c73acaef178: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20240502 08:16:39.639806 16558 raft_consensus.cc:348] T e5a2f2b6053d410b9a51b54869c45901 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.640061 16558 raft_consensus.cc:374] T e5a2f2b6053d410b9a51b54869c45901 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.640197 16558 raft_consensus.cc:729] T e5a2f2b6053d410b9a51b54869c45901 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78a635b286d46618742689bcf176adb, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.640249 16557 tablet_bootstrap.cc:492] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557: No bootstrap required, opened a new log
I20240502 08:16:39.640385 16557 ts_tablet_manager.cc:1367] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20240502 08:16:39.640520 16556 raft_consensus.cc:348] T 496e5ac0593c4ea785c3ad477076b18a P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.640555 16558 consensus_queue.cc:260] T e5a2f2b6053d410b9a51b54869c45901 P f78a635b286d46618742689bcf176adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.640714 16556 raft_consensus.cc:374] T 496e5ac0593c4ea785c3ad477076b18a P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.640861 16556 raft_consensus.cc:729] T 496e5ac0593c4ea785c3ad477076b18a P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57450ee55d64a8a87833c73acaef178, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.641214 16558 ts_tablet_manager.cc:1398] T e5a2f2b6053d410b9a51b54869c45901 P f78a635b286d46618742689bcf176adb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:39.641163 16556 consensus_queue.cc:260] T 496e5ac0593c4ea785c3ad477076b18a P e57450ee55d64a8a87833c73acaef178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.641180 16557 raft_consensus.cc:348] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.641433 16557 raft_consensus.cc:374] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.641530 16557 raft_consensus.cc:729] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc4bb2518f84cedb425773582dbe557, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.641609 16558 tablet_bootstrap.cc:492] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb: Bootstrap starting.
I20240502 08:16:39.641745 16556 ts_tablet_manager.cc:1398] T 496e5ac0593c4ea785c3ad477076b18a P e57450ee55d64a8a87833c73acaef178: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:39.641789 16557 consensus_queue.cc:260] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.642031 16556 tablet_bootstrap.cc:492] T b6db3ef6e54444309c7e3684428598a0 P e57450ee55d64a8a87833c73acaef178: Bootstrap starting.
I20240502 08:16:39.642318 16557 ts_tablet_manager.cc:1398] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:39.642613 16557 tablet_bootstrap.cc:492] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557: Bootstrap starting.
I20240502 08:16:39.643837 16556 tablet_bootstrap.cc:654] T b6db3ef6e54444309c7e3684428598a0 P e57450ee55d64a8a87833c73acaef178: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.644373 16557 tablet_bootstrap.cc:654] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.644536 16558 tablet_bootstrap.cc:654] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.652168 16556 tablet_bootstrap.cc:492] T b6db3ef6e54444309c7e3684428598a0 P e57450ee55d64a8a87833c73acaef178: No bootstrap required, opened a new log
I20240502 08:16:39.652369 16556 ts_tablet_manager.cc:1367] T b6db3ef6e54444309c7e3684428598a0 P e57450ee55d64a8a87833c73acaef178: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240502 08:16:39.653291 16557 tablet_bootstrap.cc:492] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557: No bootstrap required, opened a new log
I20240502 08:16:39.653352 16558 tablet_bootstrap.cc:492] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb: No bootstrap required, opened a new log
I20240502 08:16:39.653483 16557 ts_tablet_manager.cc:1367] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240502 08:16:39.653541 16558 ts_tablet_manager.cc:1367] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20240502 08:16:39.653286 16556 raft_consensus.cc:348] T b6db3ef6e54444309c7e3684428598a0 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.653707 16556 raft_consensus.cc:374] T b6db3ef6e54444309c7e3684428598a0 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.653829 16556 raft_consensus.cc:729] T b6db3ef6e54444309c7e3684428598a0 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57450ee55d64a8a87833c73acaef178, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.654139 16556 consensus_queue.cc:260] T b6db3ef6e54444309c7e3684428598a0 P e57450ee55d64a8a87833c73acaef178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.654219 16557 raft_consensus.cc:348] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.654316 16558 raft_consensus.cc:348] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.654491 16557 raft_consensus.cc:374] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.654525 16558 raft_consensus.cc:374] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.654594 16557 raft_consensus.cc:729] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc4bb2518f84cedb425773582dbe557, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.654613 16558 raft_consensus.cc:729] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78a635b286d46618742689bcf176adb, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.654903 16557 consensus_queue.cc:260] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.654881 16558 consensus_queue.cc:260] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.655536 16557 ts_tablet_manager.cc:1398] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:39.655678 16556 ts_tablet_manager.cc:1398] T b6db3ef6e54444309c7e3684428598a0 P e57450ee55d64a8a87833c73acaef178: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:39.655905 16557 tablet_bootstrap.cc:492] T e5a24bcd27534faaaa59a37883434723 P 1fc4bb2518f84cedb425773582dbe557: Bootstrap starting.
I20240502 08:16:39.655994 16556 tablet_bootstrap.cc:492] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178: Bootstrap starting.
I20240502 08:16:39.655969 16558 ts_tablet_manager.cc:1398] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:39.656280 16558 tablet_bootstrap.cc:492] T b6db3ef6e54444309c7e3684428598a0 P f78a635b286d46618742689bcf176adb: Bootstrap starting.
I20240502 08:16:39.658102 16556 tablet_bootstrap.cc:654] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.658380 16558 tablet_bootstrap.cc:654] T b6db3ef6e54444309c7e3684428598a0 P f78a635b286d46618742689bcf176adb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.659782 16557 tablet_bootstrap.cc:654] T e5a24bcd27534faaaa59a37883434723 P 1fc4bb2518f84cedb425773582dbe557: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.661914 16556 tablet_bootstrap.cc:492] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178: No bootstrap required, opened a new log
I20240502 08:16:39.662125 16556 ts_tablet_manager.cc:1367] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:16:39.662971 16556 raft_consensus.cc:348] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.663265 16556 raft_consensus.cc:374] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.663435 16556 raft_consensus.cc:729] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57450ee55d64a8a87833c73acaef178, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.663848 16556 consensus_queue.cc:260] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.664465 16556 ts_tablet_manager.cc:1398] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:39.664798 16556 tablet_bootstrap.cc:492] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178: Bootstrap starting.
I20240502 08:16:39.666615 16556 tablet_bootstrap.cc:654] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.669220 16557 tablet_bootstrap.cc:492] T e5a24bcd27534faaaa59a37883434723 P 1fc4bb2518f84cedb425773582dbe557: No bootstrap required, opened a new log
I20240502 08:16:39.669380 16557 ts_tablet_manager.cc:1367] T e5a24bcd27534faaaa59a37883434723 P 1fc4bb2518f84cedb425773582dbe557: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240502 08:16:39.669893 16558 tablet_bootstrap.cc:492] T b6db3ef6e54444309c7e3684428598a0 P f78a635b286d46618742689bcf176adb: No bootstrap required, opened a new log
I20240502 08:16:39.670065 16558 ts_tablet_manager.cc:1367] T b6db3ef6e54444309c7e3684428598a0 P f78a635b286d46618742689bcf176adb: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20240502 08:16:39.670373 16557 raft_consensus.cc:348] T e5a24bcd27534faaaa59a37883434723 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } }
I20240502 08:16:39.670531 16556 tablet_bootstrap.cc:492] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178: No bootstrap required, opened a new log
I20240502 08:16:39.670558 16557 raft_consensus.cc:374] T e5a24bcd27534faaaa59a37883434723 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.670671 16557 raft_consensus.cc:729] T e5a24bcd27534faaaa59a37883434723 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc4bb2518f84cedb425773582dbe557, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.670702 16556 ts_tablet_manager.cc:1367] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240502 08:16:39.670992 16557 consensus_queue.cc:260] T e5a24bcd27534faaaa59a37883434723 P 1fc4bb2518f84cedb425773582dbe557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } }
I20240502 08:16:39.670989 16558 raft_consensus.cc:348] T b6db3ef6e54444309c7e3684428598a0 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.671236 16558 raft_consensus.cc:374] T b6db3ef6e54444309c7e3684428598a0 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.671347 16558 raft_consensus.cc:729] T b6db3ef6e54444309c7e3684428598a0 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78a635b286d46618742689bcf176adb, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.671545 16557 ts_tablet_manager.cc:1398] T e5a24bcd27534faaaa59a37883434723 P 1fc4bb2518f84cedb425773582dbe557: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:39.671512 16556 raft_consensus.cc:348] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.671571 16558 consensus_queue.cc:260] T b6db3ef6e54444309c7e3684428598a0 P f78a635b286d46618742689bcf176adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.671746 16556 raft_consensus.cc:374] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.672051 16556 raft_consensus.cc:729] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57450ee55d64a8a87833c73acaef178, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.672410 16558 ts_tablet_manager.cc:1398] T b6db3ef6e54444309c7e3684428598a0 P f78a635b286d46618742689bcf176adb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:39.672346 16556 consensus_queue.cc:260] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.672645 16558 tablet_bootstrap.cc:492] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb: Bootstrap starting.
I20240502 08:16:39.672884 16556 ts_tablet_manager.cc:1398] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:39.673203 16556 tablet_bootstrap.cc:492] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178: Bootstrap starting.
I20240502 08:16:39.671881 16557 tablet_bootstrap.cc:492] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557: Bootstrap starting.
I20240502 08:16:39.674387 16558 tablet_bootstrap.cc:654] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.675472 16557 tablet_bootstrap.cc:654] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.675666 16556 tablet_bootstrap.cc:654] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.683037 16556 tablet_bootstrap.cc:492] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178: No bootstrap required, opened a new log
I20240502 08:16:39.683257 16556 ts_tablet_manager.cc:1367] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 08:16:39.684257 16556 raft_consensus.cc:348] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } }
I20240502 08:16:39.684513 16556 raft_consensus.cc:374] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.684648 16556 raft_consensus.cc:729] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57450ee55d64a8a87833c73acaef178, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.684882 16556 consensus_queue.cc:260] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } }
I20240502 08:16:39.685514 16556 ts_tablet_manager.cc:1398] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:39.685948 16556 tablet_bootstrap.cc:492] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178: Bootstrap starting.
I20240502 08:16:39.685986 16558 tablet_bootstrap.cc:492] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb: No bootstrap required, opened a new log
I20240502 08:16:39.686153 16558 ts_tablet_manager.cc:1367] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20240502 08:16:39.686725 16558 raft_consensus.cc:348] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.686931 16558 raft_consensus.cc:374] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.687021 16558 raft_consensus.cc:729] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78a635b286d46618742689bcf176adb, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.687336 16558 consensus_queue.cc:260] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.687878 16556 tablet_bootstrap.cc:654] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.687906 16558 ts_tablet_manager.cc:1398] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:39.687953 16557 tablet_bootstrap.cc:492] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557: No bootstrap required, opened a new log
I20240502 08:16:39.688113 16557 ts_tablet_manager.cc:1367] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20240502 08:16:39.689013 16558 tablet_bootstrap.cc:492] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb: Bootstrap starting.
I20240502 08:16:39.688906 16557 raft_consensus.cc:348] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.689206 16557 raft_consensus.cc:374] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.689299 16557 raft_consensus.cc:729] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc4bb2518f84cedb425773582dbe557, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.689635 16557 consensus_queue.cc:260] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.690506 16557 ts_tablet_manager.cc:1398] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:39.690730 16558 tablet_bootstrap.cc:654] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.690871 16557 tablet_bootstrap.cc:492] T e5a2f2b6053d410b9a51b54869c45901 P 1fc4bb2518f84cedb425773582dbe557: Bootstrap starting.
I20240502 08:16:39.693059 16557 tablet_bootstrap.cc:654] T e5a2f2b6053d410b9a51b54869c45901 P 1fc4bb2518f84cedb425773582dbe557: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.700186 16556 tablet_bootstrap.cc:492] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178: No bootstrap required, opened a new log
I20240502 08:16:39.700393 16556 ts_tablet_manager.cc:1367] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20240502 08:16:39.700682 16558 tablet_bootstrap.cc:492] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb: No bootstrap required, opened a new log
I20240502 08:16:39.700723 16557 tablet_bootstrap.cc:492] T e5a2f2b6053d410b9a51b54869c45901 P 1fc4bb2518f84cedb425773582dbe557: No bootstrap required, opened a new log
I20240502 08:16:39.700834 16558 ts_tablet_manager.cc:1367] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240502 08:16:39.700875 16557 ts_tablet_manager.cc:1367] T e5a2f2b6053d410b9a51b54869c45901 P 1fc4bb2518f84cedb425773582dbe557: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 08:16:39.701351 16556 raft_consensus.cc:348] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.701609 16557 raft_consensus.cc:348] T e5a2f2b6053d410b9a51b54869c45901 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.701692 16556 raft_consensus.cc:374] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.701618 16558 raft_consensus.cc:348] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.701804 16557 raft_consensus.cc:374] T e5a2f2b6053d410b9a51b54869c45901 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.701807 16556 raft_consensus.cc:729] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57450ee55d64a8a87833c73acaef178, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.701862 16558 raft_consensus.cc:374] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.701928 16557 raft_consensus.cc:729] T e5a2f2b6053d410b9a51b54869c45901 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc4bb2518f84cedb425773582dbe557, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.702001 16558 raft_consensus.cc:729] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78a635b286d46618742689bcf176adb, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.702181 16557 consensus_queue.cc:260] T e5a2f2b6053d410b9a51b54869c45901 P 1fc4bb2518f84cedb425773582dbe557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.702206 16556 consensus_queue.cc:260] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.702713 16557 ts_tablet_manager.cc:1398] T e5a2f2b6053d410b9a51b54869c45901 P 1fc4bb2518f84cedb425773582dbe557: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:39.702723 16556 ts_tablet_manager.cc:1398] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:39.702241 16558 consensus_queue.cc:260] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.703045 16557 tablet_bootstrap.cc:492] T 496e5ac0593c4ea785c3ad477076b18a P 1fc4bb2518f84cedb425773582dbe557: Bootstrap starting.
I20240502 08:16:39.705024 16557 tablet_bootstrap.cc:654] T 496e5ac0593c4ea785c3ad477076b18a P 1fc4bb2518f84cedb425773582dbe557: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.706775 16558 ts_tablet_manager.cc:1398] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:16:39.707288 16557 tablet_bootstrap.cc:492] T 496e5ac0593c4ea785c3ad477076b18a P 1fc4bb2518f84cedb425773582dbe557: No bootstrap required, opened a new log
I20240502 08:16:39.707463 16557 ts_tablet_manager.cc:1367] T 496e5ac0593c4ea785c3ad477076b18a P 1fc4bb2518f84cedb425773582dbe557: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:16:39.707491 16558 tablet_bootstrap.cc:492] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb: Bootstrap starting.
I20240502 08:16:39.708369 16557 raft_consensus.cc:348] T 496e5ac0593c4ea785c3ad477076b18a P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.708621 16557 raft_consensus.cc:374] T 496e5ac0593c4ea785c3ad477076b18a P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.708719 16557 raft_consensus.cc:729] T 496e5ac0593c4ea785c3ad477076b18a P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc4bb2518f84cedb425773582dbe557, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.708869 16558 tablet_bootstrap.cc:654] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.709012 16557 consensus_queue.cc:260] T 496e5ac0593c4ea785c3ad477076b18a P 1fc4bb2518f84cedb425773582dbe557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.709468 16557 ts_tablet_manager.cc:1398] T 496e5ac0593c4ea785c3ad477076b18a P 1fc4bb2518f84cedb425773582dbe557: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:16:39.710650 16558 tablet_bootstrap.cc:492] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb: No bootstrap required, opened a new log
I20240502 08:16:39.710785 16558 ts_tablet_manager.cc:1367] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:16:39.711406 16558 raft_consensus.cc:348] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.711596 16558 raft_consensus.cc:374] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.711722 16558 raft_consensus.cc:729] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78a635b286d46618742689bcf176adb, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.711977 16558 consensus_queue.cc:260] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.712410 16558 ts_tablet_manager.cc:1398] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:16:39.718096 16562 raft_consensus.cc:482] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.718225 16562 raft_consensus.cc:504] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.719069 16562 leader_election.cc:288] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f78a635b286d46618742689bcf176adb (127.15.81.1:44913), 1fc4bb2518f84cedb425773582dbe557 (127.15.81.3:43159)
I20240502 08:16:39.725755 16511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a2f2b6053d410b9a51b54869c45901" candidate_uuid: "e57450ee55d64a8a87833c73acaef178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc4bb2518f84cedb425773582dbe557" is_pre_election: true
I20240502 08:16:39.726128 16511 raft_consensus.cc:2454] T e5a2f2b6053d410b9a51b54869c45901 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e57450ee55d64a8a87833c73acaef178 in term 0.
I20240502 08:16:39.726089 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a2f2b6053d410b9a51b54869c45901" candidate_uuid: "e57450ee55d64a8a87833c73acaef178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78a635b286d46618742689bcf176adb" is_pre_election: true
I20240502 08:16:39.726420 16359 raft_consensus.cc:2454] T e5a2f2b6053d410b9a51b54869c45901 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e57450ee55d64a8a87833c73acaef178 in term 0.
I20240502 08:16:39.726614 16401 leader_election.cc:302] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc4bb2518f84cedb425773582dbe557, e57450ee55d64a8a87833c73acaef178; no voters:
I20240502 08:16:39.726893 16562 raft_consensus.cc:2781] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:39.726990 16562 raft_consensus.cc:482] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:39.727082 16562 raft_consensus.cc:3037] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.729130 16562 raft_consensus.cc:504] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.729707 16562 leader_election.cc:288] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [CANDIDATE]: Term 1 election: Requested vote from peers f78a635b286d46618742689bcf176adb (127.15.81.1:44913), 1fc4bb2518f84cedb425773582dbe557 (127.15.81.3:43159)
I20240502 08:16:39.730240 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a2f2b6053d410b9a51b54869c45901" candidate_uuid: "e57450ee55d64a8a87833c73acaef178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78a635b286d46618742689bcf176adb"
I20240502 08:16:39.730252 16511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a2f2b6053d410b9a51b54869c45901" candidate_uuid: "e57450ee55d64a8a87833c73acaef178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc4bb2518f84cedb425773582dbe557"
I20240502 08:16:39.730485 16511 raft_consensus.cc:3037] T e5a2f2b6053d410b9a51b54869c45901 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.730482 16359 raft_consensus.cc:3037] T e5a2f2b6053d410b9a51b54869c45901 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.732872 16511 raft_consensus.cc:2454] T e5a2f2b6053d410b9a51b54869c45901 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e57450ee55d64a8a87833c73acaef178 in term 1.
I20240502 08:16:39.732916 16359 raft_consensus.cc:2454] T e5a2f2b6053d410b9a51b54869c45901 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e57450ee55d64a8a87833c73acaef178 in term 1.
I20240502 08:16:39.733247 16401 leader_election.cc:302] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc4bb2518f84cedb425773582dbe557, e57450ee55d64a8a87833c73acaef178; no voters:
I20240502 08:16:39.733559 16562 raft_consensus.cc:2781] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:39.733971 16562 raft_consensus.cc:686] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [term 1 LEADER]: Becoming Leader. State: Replica: e57450ee55d64a8a87833c73acaef178, State: Running, Role: LEADER
I20240502 08:16:39.734287 16562 consensus_queue.cc:237] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.736894 16254 catalog_manager.cc:5484] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 reported cstate change: term changed from 0 to 1, leader changed from <none> to e57450ee55d64a8a87833c73acaef178 (127.15.81.2). New cstate: current_term: 1 leader_uuid: "e57450ee55d64a8a87833c73acaef178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:39.751847 16562 raft_consensus.cc:482] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.752099 16562 raft_consensus.cc:504] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } }
I20240502 08:16:39.752764 16562 leader_election.cc:288] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fc4bb2518f84cedb425773582dbe557 (127.15.81.3:43159), f78a635b286d46618742689bcf176adb (127.15.81.1:44913)
I20240502 08:16:39.753285 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a24bcd27534faaaa59a37883434723" candidate_uuid: "e57450ee55d64a8a87833c73acaef178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78a635b286d46618742689bcf176adb" is_pre_election: true
I20240502 08:16:39.753285 16511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a24bcd27534faaaa59a37883434723" candidate_uuid: "e57450ee55d64a8a87833c73acaef178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc4bb2518f84cedb425773582dbe557" is_pre_election: true
I20240502 08:16:39.753518 16359 raft_consensus.cc:2454] T e5a24bcd27534faaaa59a37883434723 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e57450ee55d64a8a87833c73acaef178 in term 0.
I20240502 08:16:39.753618 16511 raft_consensus.cc:2454] T e5a24bcd27534faaaa59a37883434723 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e57450ee55d64a8a87833c73acaef178 in term 0.
I20240502 08:16:39.753935 16402 leader_election.cc:302] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e57450ee55d64a8a87833c73acaef178, f78a635b286d46618742689bcf176adb; no voters:
I20240502 08:16:39.754209 16562 raft_consensus.cc:2781] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:39.754307 16562 raft_consensus.cc:482] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:39.754393 16562 raft_consensus.cc:3037] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.756336 16562 raft_consensus.cc:504] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } }
I20240502 08:16:39.756844 16562 leader_election.cc:288] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [CANDIDATE]: Term 1 election: Requested vote from peers 1fc4bb2518f84cedb425773582dbe557 (127.15.81.3:43159), f78a635b286d46618742689bcf176adb (127.15.81.1:44913)
I20240502 08:16:39.757236 16511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a24bcd27534faaaa59a37883434723" candidate_uuid: "e57450ee55d64a8a87833c73acaef178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc4bb2518f84cedb425773582dbe557"
I20240502 08:16:39.757236 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a24bcd27534faaaa59a37883434723" candidate_uuid: "e57450ee55d64a8a87833c73acaef178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78a635b286d46618742689bcf176adb"
I20240502 08:16:39.757519 16511 raft_consensus.cc:3037] T e5a24bcd27534faaaa59a37883434723 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.757544 16359 raft_consensus.cc:3037] T e5a24bcd27534faaaa59a37883434723 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.760623 16359 raft_consensus.cc:2454] T e5a24bcd27534faaaa59a37883434723 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e57450ee55d64a8a87833c73acaef178 in term 1.
I20240502 08:16:39.760623 16511 raft_consensus.cc:2454] T e5a24bcd27534faaaa59a37883434723 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e57450ee55d64a8a87833c73acaef178 in term 1.
I20240502 08:16:39.761065 16401 leader_election.cc:302] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc4bb2518f84cedb425773582dbe557, e57450ee55d64a8a87833c73acaef178; no voters:
I20240502 08:16:39.761348 16562 raft_consensus.cc:2781] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:39.761518 16562 raft_consensus.cc:686] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [term 1 LEADER]: Becoming Leader. State: Replica: e57450ee55d64a8a87833c73acaef178, State: Running, Role: LEADER
I20240502 08:16:39.761787 16562 consensus_queue.cc:237] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } }
I20240502 08:16:39.764288 16254 catalog_manager.cc:5484] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 reported cstate change: term changed from 0 to 1, leader changed from <none> to e57450ee55d64a8a87833c73acaef178 (127.15.81.2). New cstate: current_term: 1 leader_uuid: "e57450ee55d64a8a87833c73acaef178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:39.791680 16563 raft_consensus.cc:482] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.791894 16563 raft_consensus.cc:504] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.792549 16564 raft_consensus.cc:482] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.792706 16564 raft_consensus.cc:504] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.793007 16563 leader_election.cc:288] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e57450ee55d64a8a87833c73acaef178 (127.15.81.2:43791), 1fc4bb2518f84cedb425773582dbe557 (127.15.81.3:43159)
I20240502 08:16:39.793833 16564 leader_election.cc:288] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f78a635b286d46618742689bcf176adb (127.15.81.1:44913), e57450ee55d64a8a87833c73acaef178 (127.15.81.2:43791)
I20240502 08:16:39.802965 16436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b29b3bd78b247c09112e090e1144d52" candidate_uuid: "f78a635b286d46618742689bcf176adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57450ee55d64a8a87833c73acaef178" is_pre_election: true
I20240502 08:16:39.803422 16436 raft_consensus.cc:2454] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f78a635b286d46618742689bcf176adb in term 0.
I20240502 08:16:39.804271 16324 leader_election.cc:302] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e57450ee55d64a8a87833c73acaef178, f78a635b286d46618742689bcf176adb; no voters:
I20240502 08:16:39.804347 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4003407906b496b89662cfaf89d66be" candidate_uuid: "1fc4bb2518f84cedb425773582dbe557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78a635b286d46618742689bcf176adb" is_pre_election: true
I20240502 08:16:39.804567 16359 raft_consensus.cc:2454] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fc4bb2518f84cedb425773582dbe557 in term 0.
I20240502 08:16:39.804657 16563 raft_consensus.cc:2781] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:39.804795 16563 raft_consensus.cc:482] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:39.804908 16563 raft_consensus.cc:3037] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.804829 16511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b29b3bd78b247c09112e090e1144d52" candidate_uuid: "f78a635b286d46618742689bcf176adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc4bb2518f84cedb425773582dbe557" is_pre_election: true
I20240502 08:16:39.805023 16477 leader_election.cc:302] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc4bb2518f84cedb425773582dbe557, f78a635b286d46618742689bcf176adb; no voters:
I20240502 08:16:39.805308 16436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4003407906b496b89662cfaf89d66be" candidate_uuid: "1fc4bb2518f84cedb425773582dbe557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57450ee55d64a8a87833c73acaef178" is_pre_election: true
I20240502 08:16:39.805393 16564 raft_consensus.cc:2781] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:39.805518 16436 raft_consensus.cc:2454] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fc4bb2518f84cedb425773582dbe557 in term 0.
I20240502 08:16:39.805526 16564 raft_consensus.cc:482] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:39.805676 16564 raft_consensus.cc:3037] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.805060 16511 raft_consensus.cc:2454] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f78a635b286d46618742689bcf176adb in term 0.
I20240502 08:16:39.807756 16563 raft_consensus.cc:504] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.808038 16564 raft_consensus.cc:504] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.808375 16563 leader_election.cc:288] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [CANDIDATE]: Term 1 election: Requested vote from peers e57450ee55d64a8a87833c73acaef178 (127.15.81.2:43791), 1fc4bb2518f84cedb425773582dbe557 (127.15.81.3:43159)
I20240502 08:16:39.808880 16564 leader_election.cc:288] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [CANDIDATE]: Term 1 election: Requested vote from peers f78a635b286d46618742689bcf176adb (127.15.81.1:44913), e57450ee55d64a8a87833c73acaef178 (127.15.81.2:43791)
I20240502 08:16:39.808939 16436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b29b3bd78b247c09112e090e1144d52" candidate_uuid: "f78a635b286d46618742689bcf176adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57450ee55d64a8a87833c73acaef178"
I20240502 08:16:39.809131 16436 raft_consensus.cc:3037] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.809463 16435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4003407906b496b89662cfaf89d66be" candidate_uuid: "1fc4bb2518f84cedb425773582dbe557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57450ee55d64a8a87833c73acaef178"
I20240502 08:16:39.809736 16435 raft_consensus.cc:3037] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.812397 16436 raft_consensus.cc:2454] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f78a635b286d46618742689bcf176adb in term 1.
I20240502 08:16:39.808889 16511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b29b3bd78b247c09112e090e1144d52" candidate_uuid: "f78a635b286d46618742689bcf176adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc4bb2518f84cedb425773582dbe557"
I20240502 08:16:39.812721 16435 raft_consensus.cc:2454] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fc4bb2518f84cedb425773582dbe557 in term 1.
I20240502 08:16:39.812819 16511 raft_consensus.cc:3037] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.812789 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4003407906b496b89662cfaf89d66be" candidate_uuid: "1fc4bb2518f84cedb425773582dbe557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78a635b286d46618742689bcf176adb"
I20240502 08:16:39.812978 16359 raft_consensus.cc:3037] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.813104 16476 leader_election.cc:302] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc4bb2518f84cedb425773582dbe557, e57450ee55d64a8a87833c73acaef178; no voters:
I20240502 08:16:39.813395 16564 raft_consensus.cc:2781] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:39.813932 16564 raft_consensus.cc:686] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [term 1 LEADER]: Becoming Leader. State: Replica: 1fc4bb2518f84cedb425773582dbe557, State: Running, Role: LEADER
I20240502 08:16:39.814291 16564 consensus_queue.cc:237] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.815785 16324 leader_election.cc:302] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e57450ee55d64a8a87833c73acaef178, f78a635b286d46618742689bcf176adb; no voters:
I20240502 08:16:39.816134 16563 raft_consensus.cc:2781] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:39.816898 16359 raft_consensus.cc:2454] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fc4bb2518f84cedb425773582dbe557 in term 1.
I20240502 08:16:39.817287 16511 raft_consensus.cc:2454] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f78a635b286d46618742689bcf176adb in term 1.
I20240502 08:16:39.817848 16254 catalog_manager.cc:5484] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fc4bb2518f84cedb425773582dbe557 (127.15.81.3). New cstate: current_term: 1 leader_uuid: "1fc4bb2518f84cedb425773582dbe557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:39.818854 16564 raft_consensus.cc:482] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.819077 16564 raft_consensus.cc:504] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.819964 16564 leader_election.cc:288] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f78a635b286d46618742689bcf176adb (127.15.81.1:44913), e57450ee55d64a8a87833c73acaef178 (127.15.81.2:43791)
I20240502 08:16:39.820233 16564 raft_consensus.cc:482] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.820389 16564 raft_consensus.cc:504] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.820495 16435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96eecf3a17634ff39abdce6c99f4eaa6" candidate_uuid: "1fc4bb2518f84cedb425773582dbe557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57450ee55d64a8a87833c73acaef178" is_pre_election: true
I20240502 08:16:39.820724 16435 raft_consensus.cc:2454] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fc4bb2518f84cedb425773582dbe557 in term 0.
I20240502 08:16:39.820835 16563 raft_consensus.cc:686] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [term 1 LEADER]: Becoming Leader. State: Replica: f78a635b286d46618742689bcf176adb, State: Running, Role: LEADER
I20240502 08:16:39.821168 16476 leader_election.cc:302] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc4bb2518f84cedb425773582dbe557, e57450ee55d64a8a87833c73acaef178; no voters:
I20240502 08:16:39.821231 16564 leader_election.cc:288] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f78a635b286d46618742689bcf176adb (127.15.81.1:44913), e57450ee55d64a8a87833c73acaef178 (127.15.81.2:43791)
I20240502 08:16:39.821415 16564 raft_consensus.cc:2781] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:39.821539 16564 raft_consensus.cc:482] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:39.821667 16564 raft_consensus.cc:3037] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.821686 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96eecf3a17634ff39abdce6c99f4eaa6" candidate_uuid: "1fc4bb2518f84cedb425773582dbe557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78a635b286d46618742689bcf176adb" is_pre_election: true
I20240502 08:16:39.821741 16358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6db3ef6e54444309c7e3684428598a0" candidate_uuid: "1fc4bb2518f84cedb425773582dbe557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78a635b286d46618742689bcf176adb" is_pre_election: true
I20240502 08:16:39.822055 16358 raft_consensus.cc:2454] T b6db3ef6e54444309c7e3684428598a0 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fc4bb2518f84cedb425773582dbe557 in term 0.
I20240502 08:16:39.822113 16359 raft_consensus.cc:2454] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fc4bb2518f84cedb425773582dbe557 in term 0.
I20240502 08:16:39.822113 16435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6db3ef6e54444309c7e3684428598a0" candidate_uuid: "1fc4bb2518f84cedb425773582dbe557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57450ee55d64a8a87833c73acaef178" is_pre_election: true
I20240502 08:16:39.822409 16435 raft_consensus.cc:2454] T b6db3ef6e54444309c7e3684428598a0 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fc4bb2518f84cedb425773582dbe557 in term 0.
I20240502 08:16:39.822592 16477 leader_election.cc:302] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc4bb2518f84cedb425773582dbe557, f78a635b286d46618742689bcf176adb; no voters:
I20240502 08:16:39.822896 16599 raft_consensus.cc:482] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.823032 16599 raft_consensus.cc:504] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.823230 16597 raft_consensus.cc:2781] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:39.823376 16597 raft_consensus.cc:482] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:39.821174 16563 consensus_queue.cc:237] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.823635 16597 raft_consensus.cc:3037] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.824007 16599 leader_election.cc:288] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e57450ee55d64a8a87833c73acaef178 (127.15.81.2:43791), 1fc4bb2518f84cedb425773582dbe557 (127.15.81.3:43159)
I20240502 08:16:39.825044 16435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "496e5ac0593c4ea785c3ad477076b18a" candidate_uuid: "f78a635b286d46618742689bcf176adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57450ee55d64a8a87833c73acaef178" is_pre_election: true
I20240502 08:16:39.825047 16564 raft_consensus.cc:504] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.825340 16435 raft_consensus.cc:2454] T 496e5ac0593c4ea785c3ad477076b18a P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f78a635b286d46618742689bcf176adb in term 0.
I20240502 08:16:39.825965 16564 leader_election.cc:288] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [CANDIDATE]: Term 1 election: Requested vote from peers f78a635b286d46618742689bcf176adb (127.15.81.1:44913), e57450ee55d64a8a87833c73acaef178 (127.15.81.2:43791)
I20240502 08:16:39.826069 16324 leader_election.cc:302] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e57450ee55d64a8a87833c73acaef178, f78a635b286d46618742689bcf176adb; no voters:
I20240502 08:16:39.826241 16511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "496e5ac0593c4ea785c3ad477076b18a" candidate_uuid: "f78a635b286d46618742689bcf176adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc4bb2518f84cedb425773582dbe557" is_pre_election: true
I20240502 08:16:39.826314 16599 raft_consensus.cc:2781] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:39.826375 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96eecf3a17634ff39abdce6c99f4eaa6" candidate_uuid: "1fc4bb2518f84cedb425773582dbe557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78a635b286d46618742689bcf176adb"
I20240502 08:16:39.826458 16599 raft_consensus.cc:482] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:39.826571 16599 raft_consensus.cc:3037] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.826601 16359 raft_consensus.cc:3037] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.826818 16435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96eecf3a17634ff39abdce6c99f4eaa6" candidate_uuid: "1fc4bb2518f84cedb425773582dbe557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57450ee55d64a8a87833c73acaef178"
I20240502 08:16:39.826907 16597 raft_consensus.cc:504] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.827042 16435 raft_consensus.cc:3037] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.827451 16597 leader_election.cc:288] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [CANDIDATE]: Term 1 election: Requested vote from peers f78a635b286d46618742689bcf176adb (127.15.81.1:44913), e57450ee55d64a8a87833c73acaef178 (127.15.81.2:43791)
I20240502 08:16:39.827952 16436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6db3ef6e54444309c7e3684428598a0" candidate_uuid: "1fc4bb2518f84cedb425773582dbe557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57450ee55d64a8a87833c73acaef178"
I20240502 08:16:39.828192 16436 raft_consensus.cc:3037] T b6db3ef6e54444309c7e3684428598a0 P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.830036 16435 raft_consensus.cc:2454] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fc4bb2518f84cedb425773582dbe557 in term 1.
I20240502 08:16:39.829957 16599 raft_consensus.cc:504] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.830243 16436 raft_consensus.cc:2454] T b6db3ef6e54444309c7e3684428598a0 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fc4bb2518f84cedb425773582dbe557 in term 1.
I20240502 08:16:39.826467 16511 raft_consensus.cc:2454] T 496e5ac0593c4ea785c3ad477076b18a P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f78a635b286d46618742689bcf176adb in term 0.
I20240502 08:16:39.830644 16476 leader_election.cc:302] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc4bb2518f84cedb425773582dbe557, e57450ee55d64a8a87833c73acaef178; no voters:
I20240502 08:16:39.830642 16599 leader_election.cc:288] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [CANDIDATE]: Term 1 election: Requested vote from peers e57450ee55d64a8a87833c73acaef178 (127.15.81.2:43791), 1fc4bb2518f84cedb425773582dbe557 (127.15.81.3:43159)
I20240502 08:16:39.831020 16597 raft_consensus.cc:2781] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:39.831209 16597 raft_consensus.cc:686] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [term 1 LEADER]: Becoming Leader. State: Replica: 1fc4bb2518f84cedb425773582dbe557, State: Running, Role: LEADER
I20240502 08:16:39.831207 16476 leader_election.cc:302] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc4bb2518f84cedb425773582dbe557, e57450ee55d64a8a87833c73acaef178; no voters:
I20240502 08:16:39.831720 16564 raft_consensus.cc:2781] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:39.831549 16597 consensus_queue.cc:237] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.831876 16564 raft_consensus.cc:686] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [term 1 LEADER]: Becoming Leader. State: Replica: 1fc4bb2518f84cedb425773582dbe557, State: Running, Role: LEADER
I20240502 08:16:39.832165 16564 consensus_queue.cc:237] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.832679 16436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "496e5ac0593c4ea785c3ad477076b18a" candidate_uuid: "f78a635b286d46618742689bcf176adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57450ee55d64a8a87833c73acaef178"
I20240502 08:16:39.832865 16436 raft_consensus.cc:3037] T 496e5ac0593c4ea785c3ad477076b18a P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.835866 16358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6db3ef6e54444309c7e3684428598a0" candidate_uuid: "1fc4bb2518f84cedb425773582dbe557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78a635b286d46618742689bcf176adb"
I20240502 08:16:39.836114 16358 raft_consensus.cc:3037] T b6db3ef6e54444309c7e3684428598a0 P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.839169 16358 raft_consensus.cc:2454] T b6db3ef6e54444309c7e3684428598a0 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fc4bb2518f84cedb425773582dbe557 in term 1.
I20240502 08:16:39.840143 16436 raft_consensus.cc:2454] T 496e5ac0593c4ea785c3ad477076b18a P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f78a635b286d46618742689bcf176adb in term 1.
I20240502 08:16:39.840519 16324 leader_election.cc:302] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e57450ee55d64a8a87833c73acaef178, f78a635b286d46618742689bcf176adb; no voters:
I20240502 08:16:39.840811 16563 raft_consensus.cc:2781] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:39.841028 16563 raft_consensus.cc:686] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [term 1 LEADER]: Becoming Leader. State: Replica: f78a635b286d46618742689bcf176adb, State: Running, Role: LEADER
I20240502 08:16:39.841073 16359 raft_consensus.cc:2454] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fc4bb2518f84cedb425773582dbe557 in term 1.
I20240502 08:16:39.841655 16563 consensus_queue.cc:237] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.841778 16254 catalog_manager.cc:5484] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb reported cstate change: term changed from 0 to 1, leader changed from <none> to f78a635b286d46618742689bcf176adb (127.15.81.1). New cstate: current_term: 1 leader_uuid: "f78a635b286d46618742689bcf176adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:39.844094 16511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "496e5ac0593c4ea785c3ad477076b18a" candidate_uuid: "f78a635b286d46618742689bcf176adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc4bb2518f84cedb425773582dbe557"
I20240502 08:16:39.844328 16511 raft_consensus.cc:3037] T 496e5ac0593c4ea785c3ad477076b18a P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.844358 16562 raft_consensus.cc:482] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.844537 16562 raft_consensus.cc:504] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.844591 16253 catalog_manager.cc:5484] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fc4bb2518f84cedb425773582dbe557 (127.15.81.3). New cstate: current_term: 1 leader_uuid: "1fc4bb2518f84cedb425773582dbe557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:39.845274 16562 leader_election.cc:288] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f78a635b286d46618742689bcf176adb (127.15.81.1:44913), 1fc4bb2518f84cedb425773582dbe557 (127.15.81.3:43159)
I20240502 08:16:39.846997 16511 raft_consensus.cc:2454] T 496e5ac0593c4ea785c3ad477076b18a P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f78a635b286d46618742689bcf176adb in term 1.
I20240502 08:16:39.847590 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020165b5b7c743039aa2b3a5a818d17b" candidate_uuid: "e57450ee55d64a8a87833c73acaef178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78a635b286d46618742689bcf176adb" is_pre_election: true
I20240502 08:16:39.847561 16254 catalog_manager.cc:5484] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb reported cstate change: term changed from 0 to 1, leader changed from <none> to f78a635b286d46618742689bcf176adb (127.15.81.1). New cstate: current_term: 1 leader_uuid: "f78a635b286d46618742689bcf176adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:39.847903 16511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020165b5b7c743039aa2b3a5a818d17b" candidate_uuid: "e57450ee55d64a8a87833c73acaef178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc4bb2518f84cedb425773582dbe557" is_pre_election: true
I20240502 08:16:39.847900 16359 raft_consensus.cc:2454] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e57450ee55d64a8a87833c73acaef178 in term 0.
I20240502 08:16:39.848109 16511 raft_consensus.cc:2454] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e57450ee55d64a8a87833c73acaef178 in term 0.
I20240502 08:16:39.848541 16402 leader_election.cc:302] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e57450ee55d64a8a87833c73acaef178, f78a635b286d46618742689bcf176adb; no voters:
I20240502 08:16:39.848861 16562 raft_consensus.cc:2781] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:39.848990 16562 raft_consensus.cc:482] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:39.849120 16562 raft_consensus.cc:3037] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.850242 16253 catalog_manager.cc:5484] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fc4bb2518f84cedb425773582dbe557 (127.15.81.3). New cstate: current_term: 1 leader_uuid: "1fc4bb2518f84cedb425773582dbe557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:39.851958 16562 raft_consensus.cc:504] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.852648 16562 leader_election.cc:288] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [CANDIDATE]: Term 1 election: Requested vote from peers f78a635b286d46618742689bcf176adb (127.15.81.1:44913), 1fc4bb2518f84cedb425773582dbe557 (127.15.81.3:43159)
I20240502 08:16:39.853036 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020165b5b7c743039aa2b3a5a818d17b" candidate_uuid: "e57450ee55d64a8a87833c73acaef178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78a635b286d46618742689bcf176adb"
I20240502 08:16:39.853057 16511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020165b5b7c743039aa2b3a5a818d17b" candidate_uuid: "e57450ee55d64a8a87833c73acaef178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc4bb2518f84cedb425773582dbe557"
I20240502 08:16:39.853266 16359 raft_consensus.cc:3037] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.853430 16511 raft_consensus.cc:3037] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.855378 16359 raft_consensus.cc:2454] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e57450ee55d64a8a87833c73acaef178 in term 1.
I20240502 08:16:39.855556 16511 raft_consensus.cc:2454] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e57450ee55d64a8a87833c73acaef178 in term 1.
I20240502 08:16:39.855785 16402 leader_election.cc:302] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e57450ee55d64a8a87833c73acaef178, f78a635b286d46618742689bcf176adb; no voters:
I20240502 08:16:39.856144 16562 raft_consensus.cc:2781] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:39.856272 16562 raft_consensus.cc:686] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [term 1 LEADER]: Becoming Leader. State: Replica: e57450ee55d64a8a87833c73acaef178, State: Running, Role: LEADER
I20240502 08:16:39.856501 16562 consensus_queue.cc:237] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } }
I20240502 08:16:39.858986 16254 catalog_manager.cc:5484] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 reported cstate change: term changed from 0 to 1, leader changed from <none> to e57450ee55d64a8a87833c73acaef178 (127.15.81.2). New cstate: current_term: 1 leader_uuid: "e57450ee55d64a8a87833c73acaef178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:39.947971 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:39.951927 16606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:39.952551 16607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:39.953225 16609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:39.953663 15684 server_base.cc:1034] running on GCE node
I20240502 08:16:39.953954 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:39.954043 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:39.954105 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637799954086 us; error 0 us; skew 500 ppm
I20240502 08:16:39.954281 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:39.955778 15684 webserver.cc:460] Webserver started at http://127.15.81.4:42881/ using document root <none> and password file <none>
I20240502 08:16:39.956046 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:39.956143 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:39.956297 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:39.956998 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-3-root/instance:
uuid: "db460c4c916a4bca8403780422c5089d"
format_stamp: "Formatted at 2024-05-02 08:16:39 on dist-test-slave-rhmj"
I20240502 08:16:39.959777 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 08:16:39.961563 16614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:39.962005 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 08:16:39.962126 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-3-root
uuid: "db460c4c916a4bca8403780422c5089d"
format_stamp: "Formatted at 2024-05-02 08:16:39 on dist-test-slave-rhmj"
I20240502 08:16:39.962239 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714637795017386-15684-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:39.991079 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:39.991848 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:39.992520 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:39.993418 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:39.993497 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:39.993571 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:39.993628 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:40.011562 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.4:42713
I20240502 08:16:40.011628 16676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.4:42713 every 8 connection(s)
I20240502 08:16:40.019250 16677 heartbeater.cc:347] Connected to a master server at 127.15.81.62:44149
I20240502 08:16:40.019400 16677 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:40.019858 16677 heartbeater.cc:511] Master 127.15.81.62:44149 requested a full tablet report, sending...
I20240502 08:16:40.020802 16254 ts_manager.cc:194] Registered new tserver with Master: db460c4c916a4bca8403780422c5089d (127.15.81.4:42713)
I20240502 08:16:40.021689 16254 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50388
I20240502 08:16:40.188496 16562 consensus_queue.cc:1035] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.193590 16683 consensus_queue.cc:1035] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.209560 16601 consensus_queue.cc:1035] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.213812 16597 consensus_queue.cc:1035] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.222715 16310 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:16:40.225495 16310 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:16:40.225798 16310 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:16:40.258409 16563 consensus_queue.cc:1035] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.263027 16690 consensus_queue.cc:1035] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.278770 16690 consensus_queue.cc:1035] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.282254 16683 consensus_queue.cc:1035] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.282670 16597 consensus_queue.cc:1035] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.285401 16690 consensus_queue.cc:1035] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.288954 16589 consensus_queue.cc:1035] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.291898 16601 consensus_queue.cc:1035] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.307809 16601 consensus_queue.cc:1035] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.311952 16564 consensus_queue.cc:1035] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.351285 16683 consensus_queue.cc:1035] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.356063 16589 consensus_queue.cc:1035] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:41.022929 16677 heartbeater.cc:503] Master 127.15.81.62:44149 was elected leader, sending a full tablet report...
I20240502 08:16:42.226164 16310 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:16:42.227660 16310 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:16:42.227849 16310 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:16:44.228358 16310 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:16:44.229986 16310 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:16:44.230197 16310 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:16:45.237951 16510 consensus_queue.cc:237] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } attrs { replace: true } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } }
I20240502 08:16:45.241914 16436 raft_consensus.cc:1261] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Refusing update from remote peer 1fc4bb2518f84cedb425773582dbe557: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:45.242705 16706 consensus_queue.cc:1035] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:16:45.247231 16359 raft_consensus.cc:1261] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Refusing update from remote peer 1fc4bb2518f84cedb425773582dbe557: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:45.247911 16601 consensus_queue.cc:1035] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:16:45.254069 16601 raft_consensus.cc:2932] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db460c4c916a4bca8403780422c5089d (127.15.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } attrs { replace: true } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } } }
I20240502 08:16:45.256140 16359 raft_consensus.cc:2932] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db460c4c916a4bca8403780422c5089d (127.15.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } attrs { replace: true } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } } }
I20240502 08:16:45.256563 16435 raft_consensus.cc:2932] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db460c4c916a4bca8403780422c5089d (127.15.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } attrs { replace: true } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } } }
W20240502 08:16:45.262717 16475 consensus_peers.cc:485] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 -> Peer db460c4c916a4bca8403780422c5089d (127.15.81.4:42713): Couldn't send request to peer db460c4c916a4bca8403780422c5089d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4003407906b496b89662cfaf89d66be. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:45.319089 16254 catalog_manager.cc:5484] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178 reported cstate change: config changed from index -1 to 2, NON_VOTER db460c4c916a4bca8403780422c5089d (127.15.81.4) added. New cstate: current_term: 1 leader_uuid: "1fc4bb2518f84cedb425773582dbe557" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } attrs { replace: true } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } } }
I20240502 08:16:45.328660 16435 consensus_queue.cc:237] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } }
I20240502 08:16:45.330868 16359 raft_consensus.cc:1261] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Refusing update from remote peer e57450ee55d64a8a87833c73acaef178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:45.330868 16510 raft_consensus.cc:1261] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Refusing update from remote peer e57450ee55d64a8a87833c73acaef178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:45.331576 16719 consensus_queue.cc:1035] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:16:45.332234 16719 consensus_queue.cc:1035] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:16:45.334702 16702 raft_consensus.cc:2932] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db460c4c916a4bca8403780422c5089d (127.15.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } } }
I20240502 08:16:45.337749 16510 raft_consensus.cc:2932] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db460c4c916a4bca8403780422c5089d (127.15.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } } }
W20240502 08:16:45.339169 16400 consensus_peers.cc:485] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 -> Peer db460c4c916a4bca8403780422c5089d (127.15.81.4:42713): Couldn't send request to peer db460c4c916a4bca8403780422c5089d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 020165b5b7c743039aa2b3a5a818d17b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:45.339354 16359 raft_consensus.cc:2932] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db460c4c916a4bca8403780422c5089d (127.15.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } } }
I20240502 08:16:45.422390 16253 catalog_manager.cc:5484] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb reported cstate change: config changed from index -1 to 2, NON_VOTER db460c4c916a4bca8403780422c5089d (127.15.81.4) added. New cstate: current_term: 1 leader_uuid: "e57450ee55d64a8a87833c73acaef178" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } } }
I20240502 08:16:45.423460 16510 consensus_queue.cc:237] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } attrs { replace: true } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } }
I20240502 08:16:45.427884 16359 raft_consensus.cc:1261] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Refusing update from remote peer 1fc4bb2518f84cedb425773582dbe557: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:16:45.428139 16475 consensus_peers.cc:485] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 -> Peer db460c4c916a4bca8403780422c5089d (127.15.81.4:42713): Couldn't send request to peer db460c4c916a4bca8403780422c5089d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96eecf3a17634ff39abdce6c99f4eaa6. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:45.427886 16435 raft_consensus.cc:1261] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Refusing update from remote peer 1fc4bb2518f84cedb425773582dbe557: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:45.428611 16601 consensus_queue.cc:1035] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:16:45.428932 16706 consensus_queue.cc:1035] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:16:45.431689 16601 raft_consensus.cc:2932] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db460c4c916a4bca8403780422c5089d (127.15.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } attrs { replace: true } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } } }
I20240502 08:16:45.434602 16359 raft_consensus.cc:2932] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db460c4c916a4bca8403780422c5089d (127.15.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } attrs { replace: true } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } } }
I20240502 08:16:45.441682 16435 raft_consensus.cc:2932] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db460c4c916a4bca8403780422c5089d (127.15.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } attrs { replace: true } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } } }
I20240502 08:16:45.521693 16255 catalog_manager.cc:5484] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178 reported cstate change: config changed from index -1 to 2, NON_VOTER db460c4c916a4bca8403780422c5089d (127.15.81.4) added. New cstate: current_term: 1 leader_uuid: "1fc4bb2518f84cedb425773582dbe557" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } attrs { replace: true } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } } }
I20240502 08:16:45.530387 16359 consensus_queue.cc:237] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } }
I20240502 08:16:45.534219 16510 raft_consensus.cc:1261] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Refusing update from remote peer f78a635b286d46618742689bcf176adb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:45.534770 16435 raft_consensus.cc:1261] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Refusing update from remote peer f78a635b286d46618742689bcf176adb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:45.534829 16729 consensus_queue.cc:1035] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:16:45.536660 16729 consensus_queue.cc:1035] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:16:45.538887 16730 raft_consensus.cc:2932] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db460c4c916a4bca8403780422c5089d (127.15.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } } }
I20240502 08:16:45.539731 16510 raft_consensus.cc:2932] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db460c4c916a4bca8403780422c5089d (127.15.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } } }
W20240502 08:16:45.540411 16323 consensus_peers.cc:485] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb -> Peer db460c4c916a4bca8403780422c5089d (127.15.81.4:42713): Couldn't send request to peer db460c4c916a4bca8403780422c5089d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b29b3bd78b247c09112e090e1144d52. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:45.545269 16435 raft_consensus.cc:2932] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db460c4c916a4bca8403780422c5089d (127.15.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } } }
I20240502 08:16:45.619827 16253 catalog_manager.cc:5484] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178 reported cstate change: config changed from index -1 to 2, NON_VOTER db460c4c916a4bca8403780422c5089d (127.15.81.4) added. New cstate: current_term: 1 leader_uuid: "f78a635b286d46618742689bcf176adb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } } }
I20240502 08:16:45.659283 16736 ts_tablet_manager.cc:902] T c4003407906b496b89662cfaf89d66be P db460c4c916a4bca8403780422c5089d: Initiating tablet copy from peer 1fc4bb2518f84cedb425773582dbe557 (127.15.81.3:43159)
I20240502 08:16:45.661759 16736 tablet_copy_client.cc:308] T c4003407906b496b89662cfaf89d66be P db460c4c916a4bca8403780422c5089d: tablet copy: Beginning tablet copy session from remote peer at address 127.15.81.3:43159
I20240502 08:16:45.669725 16521 tablet_copy_service.cc:140] P 1fc4bb2518f84cedb425773582dbe557: Received BeginTabletCopySession request for tablet c4003407906b496b89662cfaf89d66be from peer db460c4c916a4bca8403780422c5089d ({username='slave'} at 127.0.0.1:52072)
I20240502 08:16:45.670047 16521 tablet_copy_service.cc:161] P 1fc4bb2518f84cedb425773582dbe557: Beginning new tablet copy session on tablet c4003407906b496b89662cfaf89d66be from peer db460c4c916a4bca8403780422c5089d at {username='slave'} at 127.0.0.1:52072: session id = db460c4c916a4bca8403780422c5089d-c4003407906b496b89662cfaf89d66be
I20240502 08:16:45.673533 16521 tablet_copy_source_session.cc:205] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:16:45.675437 16736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4003407906b496b89662cfaf89d66be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.682005 16736 tablet_copy_client.cc:783] T c4003407906b496b89662cfaf89d66be P db460c4c916a4bca8403780422c5089d: tablet copy: Starting download of 0 data blocks...
I20240502 08:16:45.682327 16736 tablet_copy_client.cc:647] T c4003407906b496b89662cfaf89d66be P db460c4c916a4bca8403780422c5089d: tablet copy: Starting download of 1 WAL segments...
I20240502 08:16:45.684335 16736 tablet_copy_client.cc:523] T c4003407906b496b89662cfaf89d66be P db460c4c916a4bca8403780422c5089d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:16:45.716526 16736 tablet_bootstrap.cc:492] T c4003407906b496b89662cfaf89d66be P db460c4c916a4bca8403780422c5089d: Bootstrap starting.
I20240502 08:16:45.721525 16742 ts_tablet_manager.cc:902] T 020165b5b7c743039aa2b3a5a818d17b P db460c4c916a4bca8403780422c5089d: Initiating tablet copy from peer e57450ee55d64a8a87833c73acaef178 (127.15.81.2:43791)
I20240502 08:16:45.724640 16742 tablet_copy_client.cc:308] T 020165b5b7c743039aa2b3a5a818d17b P db460c4c916a4bca8403780422c5089d: tablet copy: Beginning tablet copy session from remote peer at address 127.15.81.2:43791
I20240502 08:16:45.725646 16736 tablet_bootstrap.cc:492] T c4003407906b496b89662cfaf89d66be P db460c4c916a4bca8403780422c5089d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:16:45.726320 16736 tablet_bootstrap.cc:492] T c4003407906b496b89662cfaf89d66be P db460c4c916a4bca8403780422c5089d: Bootstrap complete.
I20240502 08:16:45.726660 16736 ts_tablet_manager.cc:1367] T c4003407906b496b89662cfaf89d66be P db460c4c916a4bca8403780422c5089d: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20240502 08:16:45.727799 16736 raft_consensus.cc:348] T c4003407906b496b89662cfaf89d66be P db460c4c916a4bca8403780422c5089d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } attrs { replace: true } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } }
I20240502 08:16:45.728111 16736 raft_consensus.cc:729] T c4003407906b496b89662cfaf89d66be P db460c4c916a4bca8403780422c5089d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db460c4c916a4bca8403780422c5089d, State: Initialized, Role: LEARNER
I20240502 08:16:45.728372 16736 consensus_queue.cc:260] T c4003407906b496b89662cfaf89d66be P db460c4c916a4bca8403780422c5089d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } attrs { replace: true } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } }
I20240502 08:16:45.729343 16736 ts_tablet_manager.cc:1398] T c4003407906b496b89662cfaf89d66be P db460c4c916a4bca8403780422c5089d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:45.730296 16521 tablet_copy_service.cc:342] P 1fc4bb2518f84cedb425773582dbe557: Request end of tablet copy session db460c4c916a4bca8403780422c5089d-c4003407906b496b89662cfaf89d66be received from {username='slave'} at 127.0.0.1:52072
I20240502 08:16:45.730479 16521 tablet_copy_service.cc:433] P 1fc4bb2518f84cedb425773582dbe557: ending tablet copy session db460c4c916a4bca8403780422c5089d-c4003407906b496b89662cfaf89d66be on tablet c4003407906b496b89662cfaf89d66be with peer db460c4c916a4bca8403780422c5089d
I20240502 08:16:45.734566 16447 tablet_copy_service.cc:140] P e57450ee55d64a8a87833c73acaef178: Received BeginTabletCopySession request for tablet 020165b5b7c743039aa2b3a5a818d17b from peer db460c4c916a4bca8403780422c5089d ({username='slave'} at 127.0.0.1:52230)
I20240502 08:16:45.734755 16447 tablet_copy_service.cc:161] P e57450ee55d64a8a87833c73acaef178: Beginning new tablet copy session on tablet 020165b5b7c743039aa2b3a5a818d17b from peer db460c4c916a4bca8403780422c5089d at {username='slave'} at 127.0.0.1:52230: session id = db460c4c916a4bca8403780422c5089d-020165b5b7c743039aa2b3a5a818d17b
I20240502 08:16:45.737269 16447 tablet_copy_source_session.cc:205] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:16:45.738423 16742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020165b5b7c743039aa2b3a5a818d17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.743800 16742 tablet_copy_client.cc:783] T 020165b5b7c743039aa2b3a5a818d17b P db460c4c916a4bca8403780422c5089d: tablet copy: Starting download of 0 data blocks...
I20240502 08:16:45.744098 16742 tablet_copy_client.cc:647] T 020165b5b7c743039aa2b3a5a818d17b P db460c4c916a4bca8403780422c5089d: tablet copy: Starting download of 1 WAL segments...
I20240502 08:16:45.746037 16742 tablet_copy_client.cc:523] T 020165b5b7c743039aa2b3a5a818d17b P db460c4c916a4bca8403780422c5089d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:16:45.816228 16742 tablet_bootstrap.cc:492] T 020165b5b7c743039aa2b3a5a818d17b P db460c4c916a4bca8403780422c5089d: Bootstrap starting.
I20240502 08:16:45.822274 16742 tablet_bootstrap.cc:492] T 020165b5b7c743039aa2b3a5a818d17b P db460c4c916a4bca8403780422c5089d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:16:45.822750 16742 tablet_bootstrap.cc:492] T 020165b5b7c743039aa2b3a5a818d17b P db460c4c916a4bca8403780422c5089d: Bootstrap complete.
I20240502 08:16:45.823001 16742 ts_tablet_manager.cc:1367] T 020165b5b7c743039aa2b3a5a818d17b P db460c4c916a4bca8403780422c5089d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 08:16:45.823832 16742 raft_consensus.cc:348] T 020165b5b7c743039aa2b3a5a818d17b P db460c4c916a4bca8403780422c5089d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } }
I20240502 08:16:45.824110 16742 raft_consensus.cc:729] T 020165b5b7c743039aa2b3a5a818d17b P db460c4c916a4bca8403780422c5089d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db460c4c916a4bca8403780422c5089d, State: Initialized, Role: LEARNER
I20240502 08:16:45.824407 16742 consensus_queue.cc:260] T 020165b5b7c743039aa2b3a5a818d17b P db460c4c916a4bca8403780422c5089d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } }
I20240502 08:16:45.825073 16742 ts_tablet_manager.cc:1398] T 020165b5b7c743039aa2b3a5a818d17b P db460c4c916a4bca8403780422c5089d: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 08:16:45.825927 16447 tablet_copy_service.cc:342] P e57450ee55d64a8a87833c73acaef178: Request end of tablet copy session db460c4c916a4bca8403780422c5089d-020165b5b7c743039aa2b3a5a818d17b received from {username='slave'} at 127.0.0.1:52230
I20240502 08:16:45.826098 16447 tablet_copy_service.cc:433] P e57450ee55d64a8a87833c73acaef178: ending tablet copy session db460c4c916a4bca8403780422c5089d-020165b5b7c743039aa2b3a5a818d17b on tablet 020165b5b7c743039aa2b3a5a818d17b with peer db460c4c916a4bca8403780422c5089d
I20240502 08:16:45.928381 16742 ts_tablet_manager.cc:902] T 96eecf3a17634ff39abdce6c99f4eaa6 P db460c4c916a4bca8403780422c5089d: Initiating tablet copy from peer 1fc4bb2518f84cedb425773582dbe557 (127.15.81.3:43159)
I20240502 08:16:45.930079 16742 tablet_copy_client.cc:308] T 96eecf3a17634ff39abdce6c99f4eaa6 P db460c4c916a4bca8403780422c5089d: tablet copy: Beginning tablet copy session from remote peer at address 127.15.81.3:43159
I20240502 08:16:45.930733 16521 tablet_copy_service.cc:140] P 1fc4bb2518f84cedb425773582dbe557: Received BeginTabletCopySession request for tablet 96eecf3a17634ff39abdce6c99f4eaa6 from peer db460c4c916a4bca8403780422c5089d ({username='slave'} at 127.0.0.1:52072)
I20240502 08:16:45.930924 16521 tablet_copy_service.cc:161] P 1fc4bb2518f84cedb425773582dbe557: Beginning new tablet copy session on tablet 96eecf3a17634ff39abdce6c99f4eaa6 from peer db460c4c916a4bca8403780422c5089d at {username='slave'} at 127.0.0.1:52072: session id = db460c4c916a4bca8403780422c5089d-96eecf3a17634ff39abdce6c99f4eaa6
I20240502 08:16:45.933820 16521 tablet_copy_source_session.cc:205] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:16:45.934988 16742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96eecf3a17634ff39abdce6c99f4eaa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.940706 16742 tablet_copy_client.cc:783] T 96eecf3a17634ff39abdce6c99f4eaa6 P db460c4c916a4bca8403780422c5089d: tablet copy: Starting download of 0 data blocks...
I20240502 08:16:45.941036 16742 tablet_copy_client.cc:647] T 96eecf3a17634ff39abdce6c99f4eaa6 P db460c4c916a4bca8403780422c5089d: tablet copy: Starting download of 1 WAL segments...
I20240502 08:16:45.942795 16742 tablet_copy_client.cc:523] T 96eecf3a17634ff39abdce6c99f4eaa6 P db460c4c916a4bca8403780422c5089d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:16:46.012842 16742 tablet_bootstrap.cc:492] T 96eecf3a17634ff39abdce6c99f4eaa6 P db460c4c916a4bca8403780422c5089d: Bootstrap starting.
I20240502 08:16:46.020016 16742 tablet_bootstrap.cc:492] T 96eecf3a17634ff39abdce6c99f4eaa6 P db460c4c916a4bca8403780422c5089d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:16:46.020406 16742 tablet_bootstrap.cc:492] T 96eecf3a17634ff39abdce6c99f4eaa6 P db460c4c916a4bca8403780422c5089d: Bootstrap complete.
I20240502 08:16:46.020665 16742 ts_tablet_manager.cc:1367] T 96eecf3a17634ff39abdce6c99f4eaa6 P db460c4c916a4bca8403780422c5089d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.001s
I20240502 08:16:46.021332 16742 raft_consensus.cc:348] T 96eecf3a17634ff39abdce6c99f4eaa6 P db460c4c916a4bca8403780422c5089d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } attrs { replace: true } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } }
I20240502 08:16:46.021600 16742 raft_consensus.cc:729] T 96eecf3a17634ff39abdce6c99f4eaa6 P db460c4c916a4bca8403780422c5089d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db460c4c916a4bca8403780422c5089d, State: Initialized, Role: LEARNER
I20240502 08:16:46.021826 16742 consensus_queue.cc:260] T 96eecf3a17634ff39abdce6c99f4eaa6 P db460c4c916a4bca8403780422c5089d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f78a635b286d46618742689bcf176adb" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 44913 } attrs { replace: true } } peers { permanent_uuid: "e57450ee55d64a8a87833c73acaef178" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43791 } } peers { permanent_uuid: "1fc4bb2518f84cedb425773582dbe557" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 43159 } } peers { permanent_uuid: "db460c4c916a4bca8403780422c5089d" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 42713 } attrs { promote: true } }
I20240502 08:16:46.022379 16742 ts_tablet_manager.cc:1398] T 96eecf3a17634ff39abdce6c99f4eaa6 P db460c4c916a4bca8403780422c5089d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:46.023316 16521 tablet_copy_service.cc:342] P 1fc4bb2518f84cedb425773582dbe557: Request end of tablet copy session db460c4c916a4bca8403780422c5089d-96eecf3a17634ff39abdce6c99f4eaa6 received from {username='slave'} at 127.0.0.1:52072
I20240502 08:16:46.023437 16521 tablet_copy_service.cc:433] P 1fc4bb2518f84cedb425773582dbe557: ending tablet copy session db460c4c916a4bca8403780422c5089d-96eecf3a17634ff39abdce6c99f4eaa6 on tablet 96eecf3a17634ff39abdce6c99f4eaa6 with peer db460c4c916a4bca8403780422c5089d
I20240502 08:16:46.039501 15684 tablet_server.cc:178] TabletServer@127.15.81.1:0 shutting down...
W20240502 08:16:46.042202 16242 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20240502 08:16:46.044808 16303 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 08:16:46.044942 16303 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 08:16:46.050168 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:16:46.050525 15684 tablet_replica.cc:330] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb: stopping tablet replica
I20240502 08:16:46.050755 15684 raft_consensus.cc:2229] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.051025 15684 raft_consensus.cc:2258] T 96eecf3a17634ff39abdce6c99f4eaa6 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.051929 15684 tablet_replica.cc:330] T b6db3ef6e54444309c7e3684428598a0 P f78a635b286d46618742689bcf176adb: stopping tablet replica
I20240502 08:16:46.052135 15684 raft_consensus.cc:2229] T b6db3ef6e54444309c7e3684428598a0 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.052321 15684 raft_consensus.cc:2258] T b6db3ef6e54444309c7e3684428598a0 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.052898 15684 tablet_replica.cc:330] T e5a24bcd27534faaaa59a37883434723 P f78a635b286d46618742689bcf176adb: stopping tablet replica
I20240502 08:16:46.053061 15684 raft_consensus.cc:2229] T e5a24bcd27534faaaa59a37883434723 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.053229 15684 raft_consensus.cc:2258] T e5a24bcd27534faaaa59a37883434723 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.053700 15684 tablet_replica.cc:330] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb: stopping tablet replica
I20240502 08:16:46.053880 15684 raft_consensus.cc:2229] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:46.054214 15684 raft_consensus.cc:2258] T 496e5ac0593c4ea785c3ad477076b18a P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.054715 15684 tablet_replica.cc:330] T e5a2f2b6053d410b9a51b54869c45901 P f78a635b286d46618742689bcf176adb: stopping tablet replica
I20240502 08:16:46.054883 15684 raft_consensus.cc:2229] T e5a2f2b6053d410b9a51b54869c45901 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.055043 15684 raft_consensus.cc:2258] T e5a2f2b6053d410b9a51b54869c45901 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.055516 15684 tablet_replica.cc:330] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb: stopping tablet replica
I20240502 08:16:46.055750 15684 raft_consensus.cc:2229] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:46.056113 15684 raft_consensus.cc:2258] T 0b29b3bd78b247c09112e090e1144d52 P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.056849 15684 tablet_replica.cc:330] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb: stopping tablet replica
I20240502 08:16:46.056998 15684 raft_consensus.cc:2229] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.057209 15684 raft_consensus.cc:2258] T c4003407906b496b89662cfaf89d66be P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.058008 15684 tablet_replica.cc:330] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb: stopping tablet replica
I20240502 08:16:46.058221 15684 raft_consensus.cc:2229] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.058439 15684 raft_consensus.cc:2258] T 020165b5b7c743039aa2b3a5a818d17b P f78a635b286d46618742689bcf176adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.075539 15684 tablet_server.cc:194] TabletServer@127.15.81.1:0 shutdown complete.
I20240502 08:16:46.080451 15684 tablet_server.cc:178] TabletServer@127.15.81.2:0 shutting down...
W20240502 08:16:46.083689 16303 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 08:16:46.083855 16303 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 08:16:46.090687 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:16:46.091009 15684 tablet_replica.cc:330] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178: stopping tablet replica
I20240502 08:16:46.091250 15684 raft_consensus.cc:2229] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.091492 15684 raft_consensus.cc:2258] T 0b29b3bd78b247c09112e090e1144d52 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.092346 15684 tablet_replica.cc:330] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178: stopping tablet replica
I20240502 08:16:46.092511 15684 raft_consensus.cc:2229] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.092711 15684 raft_consensus.cc:2258] T 96eecf3a17634ff39abdce6c99f4eaa6 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.093410 15684 tablet_replica.cc:330] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178: stopping tablet replica
I20240502 08:16:46.093585 15684 raft_consensus.cc:2229] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:46.093863 15684 raft_consensus.cc:2258] T e5a24bcd27534faaaa59a37883434723 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.094321 15684 tablet_replica.cc:330] T 496e5ac0593c4ea785c3ad477076b18a P e57450ee55d64a8a87833c73acaef178: stopping tablet replica
I20240502 08:16:46.094467 15684 raft_consensus.cc:2229] T 496e5ac0593c4ea785c3ad477076b18a P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.094626 15684 raft_consensus.cc:2258] T 496e5ac0593c4ea785c3ad477076b18a P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.095057 15684 tablet_replica.cc:330] T b6db3ef6e54444309c7e3684428598a0 P e57450ee55d64a8a87833c73acaef178: stopping tablet replica
I20240502 08:16:46.095220 15684 raft_consensus.cc:2229] T b6db3ef6e54444309c7e3684428598a0 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.095369 15684 raft_consensus.cc:2258] T b6db3ef6e54444309c7e3684428598a0 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.095849 15684 tablet_replica.cc:330] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178: stopping tablet replica
I20240502 08:16:46.095999 15684 raft_consensus.cc:2229] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:46.096375 15684 raft_consensus.cc:2258] T 020165b5b7c743039aa2b3a5a818d17b P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.097033 15684 tablet_replica.cc:330] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178: stopping tablet replica
I20240502 08:16:46.097198 15684 raft_consensus.cc:2229] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.097378 15684 raft_consensus.cc:2258] T c4003407906b496b89662cfaf89d66be P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.098107 15684 tablet_replica.cc:330] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178: stopping tablet replica
I20240502 08:16:46.098304 15684 raft_consensus.cc:2229] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:46.098605 15684 raft_consensus.cc:2258] T e5a2f2b6053d410b9a51b54869c45901 P e57450ee55d64a8a87833c73acaef178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.104635 15684 tablet_server.cc:194] TabletServer@127.15.81.2:0 shutdown complete.
I20240502 08:16:46.109395 15684 tablet_server.cc:178] TabletServer@127.15.81.3:0 shutting down...
W20240502 08:16:46.111482 16303 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 08:16:46.111598 16303 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 08:16:46.113634 16303 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 08:16:46.113955 16303 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 08:16:46.121078 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:16:46.121321 15684 tablet_replica.cc:330] T e5a2f2b6053d410b9a51b54869c45901 P 1fc4bb2518f84cedb425773582dbe557: stopping tablet replica
I20240502 08:16:46.121536 15684 raft_consensus.cc:2229] T e5a2f2b6053d410b9a51b54869c45901 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.121749 15684 raft_consensus.cc:2258] T e5a2f2b6053d410b9a51b54869c45901 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.122328 15684 tablet_replica.cc:330] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557: stopping tablet replica
I20240502 08:16:46.122506 15684 raft_consensus.cc:2229] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:46.122850 15684 raft_consensus.cc:2258] T 96eecf3a17634ff39abdce6c99f4eaa6 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.123569 15684 tablet_replica.cc:330] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557: stopping tablet replica
I20240502 08:16:46.123751 15684 raft_consensus.cc:2229] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.123939 15684 raft_consensus.cc:2258] T 0b29b3bd78b247c09112e090e1144d52 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.124562 15684 tablet_replica.cc:330] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557: stopping tablet replica
I20240502 08:16:46.124719 15684 raft_consensus.cc:2229] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:46.124958 15684 raft_consensus.cc:2258] T b6db3ef6e54444309c7e3684428598a0 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.125401 15684 tablet_replica.cc:330] T 496e5ac0593c4ea785c3ad477076b18a P 1fc4bb2518f84cedb425773582dbe557: stopping tablet replica
I20240502 08:16:46.125555 15684 raft_consensus.cc:2229] T 496e5ac0593c4ea785c3ad477076b18a P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.125701 15684 raft_consensus.cc:2258] T 496e5ac0593c4ea785c3ad477076b18a P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.126098 15684 tablet_replica.cc:330] T e5a24bcd27534faaaa59a37883434723 P 1fc4bb2518f84cedb425773582dbe557: stopping tablet replica
I20240502 08:16:46.126245 15684 raft_consensus.cc:2229] T e5a24bcd27534faaaa59a37883434723 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.126376 15684 raft_consensus.cc:2258] T e5a24bcd27534faaaa59a37883434723 P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.126791 15684 tablet_replica.cc:330] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557: stopping tablet replica
I20240502 08:16:46.126926 15684 raft_consensus.cc:2229] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.127100 15684 raft_consensus.cc:2258] T 020165b5b7c743039aa2b3a5a818d17b P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.127771 15684 tablet_replica.cc:330] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557: stopping tablet replica
I20240502 08:16:46.127929 15684 raft_consensus.cc:2229] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:46.128237 15684 raft_consensus.cc:2258] T c4003407906b496b89662cfaf89d66be P 1fc4bb2518f84cedb425773582dbe557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.143642 15684 tablet_server.cc:194] TabletServer@127.15.81.3:0 shutdown complete.
I20240502 08:16:46.147938 15684 tablet_server.cc:178] TabletServer@127.15.81.4:0 shutting down...
I20240502 08:16:46.156955 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:16:46.157495 15684 tablet_replica.cc:330] T 96eecf3a17634ff39abdce6c99f4eaa6 P db460c4c916a4bca8403780422c5089d: stopping tablet replica
I20240502 08:16:46.157747 15684 raft_consensus.cc:2229] T 96eecf3a17634ff39abdce6c99f4eaa6 P db460c4c916a4bca8403780422c5089d [term 1 LEARNER]: Raft consensus shutting down.
I20240502 08:16:46.157919 15684 raft_consensus.cc:2258] T 96eecf3a17634ff39abdce6c99f4eaa6 P db460c4c916a4bca8403780422c5089d [term 1 LEARNER]: Raft consensus is shut down!
I20240502 08:16:46.158567 15684 tablet_replica.cc:330] T c4003407906b496b89662cfaf89d66be P db460c4c916a4bca8403780422c5089d: stopping tablet replica
I20240502 08:16:46.158689 15684 raft_consensus.cc:2229] T c4003407906b496b89662cfaf89d66be P db460c4c916a4bca8403780422c5089d [term 1 LEARNER]: Raft consensus shutting down.
I20240502 08:16:46.158828 15684 raft_consensus.cc:2258] T c4003407906b496b89662cfaf89d66be P db460c4c916a4bca8403780422c5089d [term 1 LEARNER]: Raft consensus is shut down!
I20240502 08:16:46.159302 15684 tablet_replica.cc:330] T 020165b5b7c743039aa2b3a5a818d17b P db460c4c916a4bca8403780422c5089d: stopping tablet replica
I20240502 08:16:46.159418 15684 raft_consensus.cc:2229] T 020165b5b7c743039aa2b3a5a818d17b P db460c4c916a4bca8403780422c5089d [term 1 LEARNER]: Raft consensus shutting down.
I20240502 08:16:46.159525 15684 raft_consensus.cc:2258] T 020165b5b7c743039aa2b3a5a818d17b P db460c4c916a4bca8403780422c5089d [term 1 LEARNER]: Raft consensus is shut down!
I20240502 08:16:46.173228 15684 tablet_server.cc:194] TabletServer@127.15.81.4:0 shutdown complete.
I20240502 08:16:46.176393 15684 master.cc:537] Master@127.15.81.62:44149 shutting down...
I20240502 08:16:46.183004 15684 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:46.183209 15684 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.183339 15684 tablet_replica.cc:330] T 00000000000000000000000000000000 P cdf6995650b043faa686c00a8280fc8d: stopping tablet replica
I20240502 08:16:46.197206 15684 master.cc:552] Master@127.15.81.62:44149 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (8052 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240502 08:16:46.215476 15684 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.81.62:39313,127.15.81.61:36075,127.15.81.60:42167
I20240502 08:16:46.216149 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:46.219720 16759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:46.220479 16760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:46.222374 15684 server_base.cc:1034] running on GCE node
W20240502 08:16:46.222823 16762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:46.223169 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:46.223250 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:46.223323 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637806223302 us; error 0 us; skew 500 ppm
I20240502 08:16:46.223510 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:46.224884 15684 webserver.cc:460] Webserver started at http://127.15.81.62:46355/ using document root <none> and password file <none>
I20240502 08:16:46.225139 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:46.225245 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:46.225397 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:46.226086 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-0-root/instance:
uuid: "ac18cf8af21c40eebbe3aad070aeffac"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-rhmj"
I20240502 08:16:46.229015 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 08:16:46.230755 16767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.231117 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240502 08:16:46.231254 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-0-root
uuid: "ac18cf8af21c40eebbe3aad070aeffac"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-rhmj"
I20240502 08:16:46.231377 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:46.251775 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:46.252369 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:46.266475 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.62:39313
I20240502 08:16:46.266494 16818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.62:39313 every 8 connection(s)
I20240502 08:16:46.267853 16819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.267870 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:16:46.270812 16819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } has no permanent_uuid. Determining permanent_uuid...
W20240502 08:16:46.271395 16821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:46.272567 16822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:46.274297 15684 server_base.cc:1034] running on GCE node
W20240502 08:16:46.275027 16826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:46.275408 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:46.275496 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:46.275566 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637806275542 us; error 0 us; skew 500 ppm
I20240502 08:16:46.275800 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:46.277307 15684 webserver.cc:460] Webserver started at http://127.15.81.61:33029/ using document root <none> and password file <none>
I20240502 08:16:46.277590 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:46.277699 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:46.277876 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:46.278614 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-1-root/instance:
uuid: "1603b6ada9fb44f796adc27703e340af"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-rhmj"
I20240502 08:16:46.279386 16819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } has no permanent_uuid. Determining permanent_uuid...
W20240502 08:16:46.281415 16819 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.15.81.61:36075: Network error: Client connection negotiation failed: client connection to 127.15.81.61:36075: connect: Connection refused (error 111)
I20240502 08:16:46.281600 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 08:16:46.283186 16832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.283550 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 08:16:46.283708 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-1-root
uuid: "1603b6ada9fb44f796adc27703e340af"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-rhmj"
I20240502 08:16:46.283834 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:46.302049 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:46.302644 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:46.319026 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.61:36075
I20240502 08:16:46.319123 16884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.61:36075 every 8 connection(s)
I20240502 08:16:46.320729 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:16:46.320850 16885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:46.324282 16887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:46.324697 16885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } has no permanent_uuid. Determining permanent_uuid...
W20240502 08:16:46.325270 16888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:46.326018 16890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:46.326727 15684 server_base.cc:1034] running on GCE node
I20240502 08:16:46.327248 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:46.327358 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:46.327422 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637806327398 us; error 0 us; skew 500 ppm
I20240502 08:16:46.327646 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:46.329247 15684 webserver.cc:460] Webserver started at http://127.15.81.60:38167/ using document root <none> and password file <none>
I20240502 08:16:46.329540 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:46.329644 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:46.329835 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:46.330703 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-2-root/instance:
uuid: "55ab5fa0c46c459d963e41d854829123"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-rhmj"
I20240502 08:16:46.332101 16885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:46.334059 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 08:16:46.335673 16819 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } attempt: 1
I20240502 08:16:46.337244 16897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.338603 15684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 08:16:46.338734 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-2-root
uuid: "55ab5fa0c46c459d963e41d854829123"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-rhmj"
I20240502 08:16:46.338853 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:46.340158 16885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:46.346202 16819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } has no permanent_uuid. Determining permanent_uuid...
W20240502 08:16:46.348712 16819 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.15.81.60:42167: Network error: Client connection negotiation failed: client connection to 127.15.81.60:42167: connect: Connection refused (error 111)
W20240502 08:16:46.352727 16885 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.15.81.60:42167: Network error: Client connection negotiation failed: client connection to 127.15.81.60:42167: connect: Connection refused (error 111)
I20240502 08:16:46.361038 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:46.361625 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:46.373785 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.60:42167
I20240502 08:16:46.373831 16951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.60:42167 every 8 connection(s)
I20240502 08:16:46.374769 15684 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:16:46.375293 16952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.377673 16952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:46.378938 16885 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } attempt: 1
I20240502 08:16:46.385378 16952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:46.387920 16885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af: Bootstrap starting.
I20240502 08:16:46.388963 16819 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } attempt: 1
I20240502 08:16:46.389454 16885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.391846 16885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af: No bootstrap required, opened a new log
I20240502 08:16:46.392783 16885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } }
I20240502 08:16:46.393035 16885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.393169 16885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1603b6ada9fb44f796adc27703e340af, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.393448 16885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } }
I20240502 08:16:46.394299 16957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } } }
I20240502 08:16:46.394558 16885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:46.394587 16957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:46.395017 16952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:46.400735 16819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac: Bootstrap starting.
I20240502 08:16:46.403069 16819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:46.404675 16968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240502 08:16:46.404765 16952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123: Bootstrap starting.
W20240502 08:16:46.404871 16968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 08:16:46.405109 16819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac: No bootstrap required, opened a new log
I20240502 08:16:46.405778 16819 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } }
I20240502 08:16:46.405966 16819 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.406049 16819 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac18cf8af21c40eebbe3aad070aeffac, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.406334 16952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.406296 16819 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } }
I20240502 08:16:46.406956 16970 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } } }
I20240502 08:16:46.407277 16819 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:46.407275 16970 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:46.408403 16952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123: No bootstrap required, opened a new log
I20240502 08:16:46.409147 16952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } }
I20240502 08:16:46.409462 16952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.409569 16952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ab5fa0c46c459d963e41d854829123, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.409855 16952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } }
I20240502 08:16:46.410627 16977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } } }
I20240502 08:16:46.411032 16952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:46.411007 16977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:46.412338 15684 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240502 08:16:46.412436 15684 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240502 08:16:46.413405 16984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 08:16:46.413506 16984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 08:16:46.416474 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:46.416786 16995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 08:16:46.416885 16995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 08:16:46.422142 16997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:46.422811 16996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:46.423405 16999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:46.424330 15684 server_base.cc:1034] running on GCE node
I20240502 08:16:46.424587 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:46.424657 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:46.424734 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637806424699 us; error 0 us; skew 500 ppm
I20240502 08:16:46.424897 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:46.426920 15684 webserver.cc:460] Webserver started at http://127.15.81.1:46121/ using document root <none> and password file <none>
I20240502 08:16:46.427142 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:46.427237 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:46.427374 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:46.428114 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-0-root/instance:
uuid: "5f3106fa1236489f900e1f0a33cd63ca"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-rhmj"
I20240502 08:16:46.430577 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:16:46.432395 17004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.432765 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:16:46.432870 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-0-root
uuid: "5f3106fa1236489f900e1f0a33cd63ca"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-rhmj"
I20240502 08:16:46.433001 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:46.456758 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:46.457434 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:46.458074 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:46.459079 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:46.459149 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.459264 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:46.459319 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.476421 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.1:45733
I20240502 08:16:46.476446 17067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.1:45733 every 8 connection(s)
I20240502 08:16:46.483286 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:46.488855 17075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:46.490494 17068 heartbeater.cc:347] Connected to a master server at 127.15.81.60:42167
I20240502 08:16:46.490767 17068 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:46.490922 17069 heartbeater.cc:347] Connected to a master server at 127.15.81.62:39313
I20240502 08:16:46.491067 17069 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:46.491640 17069 heartbeater.cc:511] Master 127.15.81.62:39313 requested a full tablet report, sending...
I20240502 08:16:46.492200 17070 heartbeater.cc:347] Connected to a master server at 127.15.81.61:36075
I20240502 08:16:46.492349 17070 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:46.492607 17068 heartbeater.cc:511] Master 127.15.81.60:42167 requested a full tablet report, sending...
I20240502 08:16:46.492851 17070 heartbeater.cc:511] Master 127.15.81.61:36075 requested a full tablet report, sending...
I20240502 08:16:46.493110 16784 ts_manager.cc:194] Registered new tserver with Master: 5f3106fa1236489f900e1f0a33cd63ca (127.15.81.1:45733)
I20240502 08:16:46.493980 16850 ts_manager.cc:194] Registered new tserver with Master: 5f3106fa1236489f900e1f0a33cd63ca (127.15.81.1:45733)
I20240502 08:16:46.494066 16916 ts_manager.cc:194] Registered new tserver with Master: 5f3106fa1236489f900e1f0a33cd63ca (127.15.81.1:45733)
W20240502 08:16:46.495857 17076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:46.498315 17078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:46.499229 15684 server_base.cc:1034] running on GCE node
I20240502 08:16:46.499496 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:46.499589 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:46.499671 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637806499651 us; error 0 us; skew 500 ppm
I20240502 08:16:46.499858 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:46.501186 15684 webserver.cc:460] Webserver started at http://127.15.81.2:43575/ using document root <none> and password file <none>
I20240502 08:16:46.501436 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:46.501521 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:46.501669 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:46.502331 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-1-root/instance:
uuid: "6167a8d5011843e4af9c48a0a757ad23"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-rhmj"
I20240502 08:16:46.505148 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 08:16:46.506698 17083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.507079 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240502 08:16:46.507231 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-1-root
uuid: "6167a8d5011843e4af9c48a0a757ad23"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-rhmj"
I20240502 08:16:46.507362 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:46.524446 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:46.525058 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:46.525693 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:46.526602 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:46.526693 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.526805 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:46.526882 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.545789 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.2:43787
I20240502 08:16:46.545866 17146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.2:43787 every 8 connection(s)
I20240502 08:16:46.548360 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:16:46.558113 17149 heartbeater.cc:347] Connected to a master server at 127.15.81.61:36075
I20240502 08:16:46.558202 17147 heartbeater.cc:347] Connected to a master server at 127.15.81.60:42167
I20240502 08:16:46.558329 17149 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:46.558328 17147 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:46.558930 17149 heartbeater.cc:511] Master 127.15.81.61:36075 requested a full tablet report, sending...
I20240502 08:16:46.559384 17147 heartbeater.cc:511] Master 127.15.81.60:42167 requested a full tablet report, sending...
I20240502 08:16:46.560245 16850 ts_manager.cc:194] Registered new tserver with Master: 6167a8d5011843e4af9c48a0a757ad23 (127.15.81.2:43787)
W20240502 08:16:46.560257 17154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:46.561518 16916 ts_manager.cc:194] Registered new tserver with Master: 6167a8d5011843e4af9c48a0a757ad23 (127.15.81.2:43787)
W20240502 08:16:46.562837 17156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:46.567093 17148 heartbeater.cc:347] Connected to a master server at 127.15.81.62:39313
I20240502 08:16:46.567281 17148 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:46.567979 17148 heartbeater.cc:511] Master 127.15.81.62:39313 requested a full tablet report, sending...
I20240502 08:16:46.569057 16784 ts_manager.cc:194] Registered new tserver with Master: 6167a8d5011843e4af9c48a0a757ad23 (127.15.81.2:43787)
W20240502 08:16:46.569923 17158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:46.570966 15684 server_base.cc:1034] running on GCE node
I20240502 08:16:46.571280 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:46.571347 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:46.571420 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637806571386 us; error 0 us; skew 500 ppm
I20240502 08:16:46.571566 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:46.572952 15684 webserver.cc:460] Webserver started at http://127.15.81.3:41449/ using document root <none> and password file <none>
I20240502 08:16:46.573151 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:46.573247 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:46.573360 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:46.574007 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-2-root/instance:
uuid: "1c87426d93224befaac733a622db504d"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-rhmj"
I20240502 08:16:46.576469 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:16:46.578167 17163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.578514 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 08:16:46.578627 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-2-root
uuid: "1c87426d93224befaac733a622db504d"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-rhmj"
I20240502 08:16:46.578779 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714637795017386-15684-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:46.594609 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:46.595961 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:46.596532 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:46.597429 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:46.597499 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.597661 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:46.597743 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.615152 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.3:35745
I20240502 08:16:46.615176 17225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.3:35745 every 8 connection(s)
I20240502 08:16:46.626515 17227 heartbeater.cc:347] Connected to a master server at 127.15.81.62:39313
I20240502 08:16:46.626699 17227 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:46.627329 17227 heartbeater.cc:511] Master 127.15.81.62:39313 requested a full tablet report, sending...
I20240502 08:16:46.627960 17226 heartbeater.cc:347] Connected to a master server at 127.15.81.60:42167
I20240502 08:16:46.628110 17226 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:46.628232 17228 heartbeater.cc:347] Connected to a master server at 127.15.81.61:36075
I20240502 08:16:46.628366 17228 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:46.628726 17226 heartbeater.cc:511] Master 127.15.81.60:42167 requested a full tablet report, sending...
I20240502 08:16:46.628813 17228 heartbeater.cc:511] Master 127.15.81.61:36075 requested a full tablet report, sending...
I20240502 08:16:46.629451 16784 ts_manager.cc:194] Registered new tserver with Master: 1c87426d93224befaac733a622db504d (127.15.81.3:35745)
I20240502 08:16:46.629709 16916 ts_manager.cc:194] Registered new tserver with Master: 1c87426d93224befaac733a622db504d (127.15.81.3:35745)
I20240502 08:16:46.629837 16850 ts_manager.cc:194] Registered new tserver with Master: 1c87426d93224befaac733a622db504d (127.15.81.3:35745)
I20240502 08:16:46.630270 15684 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.010968254s
I20240502 08:16:46.681807 16977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.682049 16977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } }
I20240502 08:16:46.682780 16977 leader_election.cc:288] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac18cf8af21c40eebbe3aad070aeffac (127.15.81.62:39313), 1603b6ada9fb44f796adc27703e340af (127.15.81.61:36075)
I20240502 08:16:46.683524 16794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55ab5fa0c46c459d963e41d854829123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac18cf8af21c40eebbe3aad070aeffac" is_pre_election: true
I20240502 08:16:46.683552 16860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55ab5fa0c46c459d963e41d854829123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1603b6ada9fb44f796adc27703e340af" is_pre_election: true
I20240502 08:16:46.683863 16860 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55ab5fa0c46c459d963e41d854829123 in term 0.
I20240502 08:16:46.683868 16794 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55ab5fa0c46c459d963e41d854829123 in term 0.
I20240502 08:16:46.684316 16903 leader_election.cc:302] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55ab5fa0c46c459d963e41d854829123, ac18cf8af21c40eebbe3aad070aeffac; no voters:
I20240502 08:16:46.684598 16977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:46.684697 16977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:46.684787 16977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.718218 16977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } }
I20240502 08:16:46.719076 16977 leader_election.cc:288] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [CANDIDATE]: Term 1 election: Requested vote from peers ac18cf8af21c40eebbe3aad070aeffac (127.15.81.62:39313), 1603b6ada9fb44f796adc27703e340af (127.15.81.61:36075)
I20240502 08:16:46.719707 16860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55ab5fa0c46c459d963e41d854829123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1603b6ada9fb44f796adc27703e340af"
I20240502 08:16:46.719748 16794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55ab5fa0c46c459d963e41d854829123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac18cf8af21c40eebbe3aad070aeffac"
I20240502 08:16:46.719960 16860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.719993 16794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.722189 16860 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55ab5fa0c46c459d963e41d854829123 in term 1.
I20240502 08:16:46.722702 16900 leader_election.cc:302] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1603b6ada9fb44f796adc27703e340af, 55ab5fa0c46c459d963e41d854829123; no voters:
I20240502 08:16:46.723037 16977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:46.723459 16977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [term 1 LEADER]: Becoming Leader. State: Replica: 55ab5fa0c46c459d963e41d854829123, State: Running, Role: LEADER
I20240502 08:16:46.723759 16977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } }
I20240502 08:16:46.725142 17234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55ab5fa0c46c459d963e41d854829123. Latest consensus state: current_term: 1 leader_uuid: "55ab5fa0c46c459d963e41d854829123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } } }
I20240502 08:16:46.725493 17234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:46.726130 17236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:16:46.727532 17236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:16:46.730409 16860 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 1 FOLLOWER]: Refusing update from remote peer 55ab5fa0c46c459d963e41d854829123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:46.730943 16977 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:46.733296 16957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55ab5fa0c46c459d963e41d854829123. Latest consensus state: current_term: 1 leader_uuid: "55ab5fa0c46c459d963e41d854829123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } } }
I20240502 08:16:46.733588 16957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:46.734910 17234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55ab5fa0c46c459d963e41d854829123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } } }
I20240502 08:16:46.735267 17234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:46.735569 16957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55ab5fa0c46c459d963e41d854829123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } } }
I20240502 08:16:46.735819 16957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:46.735982 17236 catalog_manager.cc:1347] Generated new cluster ID: 4ce64d649e4f42a39e32cf5a121039c7
I20240502 08:16:46.736060 17236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:16:46.760702 17236 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:16:46.761754 17236 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:16:46.775187 17236 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123: Generated new TSK 0
I20240502 08:16:46.775511 17236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:16:46.818992 16794 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55ab5fa0c46c459d963e41d854829123 in term 1.
I20240502 08:16:46.819708 16793 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [term 1 FOLLOWER]: Refusing update from remote peer 55ab5fa0c46c459d963e41d854829123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:46.820765 17234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:46.826118 16970 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55ab5fa0c46c459d963e41d854829123. Latest consensus state: current_term: 1 leader_uuid: "55ab5fa0c46c459d963e41d854829123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } } }
I20240502 08:16:46.826383 16970 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:46.826226 17244 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55ab5fa0c46c459d963e41d854829123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } } }
I20240502 08:16:46.826537 17244 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:47.199890 17234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55ab5fa0c46c459d963e41d854829123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } } }
I20240502 08:16:47.200285 17234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:47.406239 16968 catalog_manager.cc:1259] Loaded cluster ID: 4ce64d649e4f42a39e32cf5a121039c7
I20240502 08:16:47.406364 16968 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af: loading cluster ID for follower catalog manager: success
I20240502 08:16:47.408393 16968 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af: acquiring CA information for follower catalog manager: success
I20240502 08:16:47.409449 16968 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 08:16:47.414700 16984 catalog_manager.cc:1259] Loaded cluster ID: 4ce64d649e4f42a39e32cf5a121039c7
I20240502 08:16:47.414791 16984 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac: loading cluster ID for follower catalog manager: success
I20240502 08:16:47.416627 16984 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac: acquiring CA information for follower catalog manager: success
I20240502 08:16:47.417620 16984 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 08:16:47.496646 16916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35876
I20240502 08:16:47.563882 16916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35888
I20240502 08:16:47.632122 16916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35894
I20240502 08:16:48.415654 16994 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:16:48.497864 17068 heartbeater.cc:503] Master 127.15.81.60:42167 was elected leader, sending a full tablet report...
I20240502 08:16:48.565052 17147 heartbeater.cc:503] Master 127.15.81.60:42167 was elected leader, sending a full tablet report...
I20240502 08:16:48.633303 17226 heartbeater.cc:503] Master 127.15.81.60:42167 was elected leader, sending a full tablet report...
I20240502 08:16:48.882611 15684 test_util.cc:268] Using random seed: 964973438
I20240502 08:16:48.910954 16916 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59368:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:48.947254 17033 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea63328a6ea4f3cb4836a81081d48b4 (DEFAULT_TABLE table=test-workload [id=747d1d3e490e4d89b89e5aee06a9f3eb]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 08:16:48.948025 17033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea63328a6ea4f3cb4836a81081d48b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.954944 17191 tablet_service.cc:1467] Processing CreateTablet for tablet 40c25833f5164019a3ca32dd9b9e9a7a (DEFAULT_TABLE table=test-workload [id=747d1d3e490e4d89b89e5aee06a9f3eb]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 08:16:48.954897 17111 tablet_service.cc:1467] Processing CreateTablet for tablet 575c5504df8e44fbac3a8c15d2d818be (DEFAULT_TABLE table=test-workload [id=747d1d3e490e4d89b89e5aee06a9f3eb]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 08:16:48.955070 17277 tablet_bootstrap.cc:492] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca: Bootstrap starting.
I20240502 08:16:48.955516 17111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 575c5504df8e44fbac3a8c15d2d818be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.955518 17191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c25833f5164019a3ca32dd9b9e9a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.956854 17277 tablet_bootstrap.cc:654] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.962260 17277 tablet_bootstrap.cc:492] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca: No bootstrap required, opened a new log
I20240502 08:16:48.962443 17277 ts_tablet_manager.cc:1367] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 08:16:48.963210 17280 tablet_bootstrap.cc:492] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23: Bootstrap starting.
I20240502 08:16:48.963567 17277 raft_consensus.cc:348] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f3106fa1236489f900e1f0a33cd63ca" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 45733 } }
I20240502 08:16:48.963812 17277 raft_consensus.cc:374] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.963927 17277 raft_consensus.cc:729] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f3106fa1236489f900e1f0a33cd63ca, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.964099 17279 tablet_bootstrap.cc:492] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d: Bootstrap starting.
I20240502 08:16:48.964260 17277 consensus_queue.cc:260] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f3106fa1236489f900e1f0a33cd63ca" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 45733 } }
I20240502 08:16:48.964529 17277 raft_consensus.cc:388] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:48.964624 17277 raft_consensus.cc:482] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:48.964759 17277 raft_consensus.cc:3037] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.965131 17280 tablet_bootstrap.cc:654] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.966089 17279 tablet_bootstrap.cc:654] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.967109 17280 tablet_bootstrap.cc:492] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23: No bootstrap required, opened a new log
I20240502 08:16:48.967255 17280 ts_tablet_manager.cc:1367] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:48.967870 17277 raft_consensus.cc:504] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f3106fa1236489f900e1f0a33cd63ca" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 45733 } }
I20240502 08:16:48.968127 17280 raft_consensus.cc:348] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6167a8d5011843e4af9c48a0a757ad23" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43787 } }
I20240502 08:16:48.968161 17277 leader_election.cc:302] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f3106fa1236489f900e1f0a33cd63ca; no voters:
I20240502 08:16:48.968300 17280 raft_consensus.cc:374] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.968456 17280 raft_consensus.cc:729] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6167a8d5011843e4af9c48a0a757ad23, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.968497 17279 tablet_bootstrap.cc:492] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d: No bootstrap required, opened a new log
I20240502 08:16:48.968694 17279 ts_tablet_manager.cc:1367] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 08:16:48.968730 17280 consensus_queue.cc:260] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6167a8d5011843e4af9c48a0a757ad23" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43787 } }
I20240502 08:16:48.968968 17280 raft_consensus.cc:388] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:48.969072 17280 raft_consensus.cc:482] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:48.969194 17280 raft_consensus.cc:3037] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.969728 17279 raft_consensus.cc:348] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c87426d93224befaac733a622db504d" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 35745 } }
I20240502 08:16:48.969966 17279 raft_consensus.cc:374] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.970104 17279 raft_consensus.cc:729] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c87426d93224befaac733a622db504d, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.970391 17279 consensus_queue.cc:260] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c87426d93224befaac733a622db504d" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 35745 } }
I20240502 08:16:48.970639 17279 raft_consensus.cc:388] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:48.970767 17279 raft_consensus.cc:482] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:48.970918 17279 raft_consensus.cc:3037] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.971624 17280 raft_consensus.cc:504] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6167a8d5011843e4af9c48a0a757ad23" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43787 } }
I20240502 08:16:48.971860 17280 leader_election.cc:302] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6167a8d5011843e4af9c48a0a757ad23; no voters:
I20240502 08:16:48.972373 17280 leader_election.cc:288] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:48.972623 17284 raft_consensus.cc:2781] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:48.974844 17277 leader_election.cc:288] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:48.974818 17279 raft_consensus.cc:504] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c87426d93224befaac733a622db504d" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 35745 } }
I20240502 08:16:48.975123 17279 leader_election.cc:302] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c87426d93224befaac733a622db504d; no voters:
I20240502 08:16:48.975998 17280 ts_tablet_manager.cc:1398] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240502 08:16:48.976276 17283 raft_consensus.cc:2781] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:48.976794 17284 raft_consensus.cc:686] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23 [term 1 LEADER]: Becoming Leader. State: Replica: 6167a8d5011843e4af9c48a0a757ad23, State: Running, Role: LEADER
I20240502 08:16:48.977169 17279 leader_election.cc:288] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:48.977394 17289 raft_consensus.cc:2781] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:48.977466 17284 consensus_queue.cc:237] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6167a8d5011843e4af9c48a0a757ad23" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43787 } }
I20240502 08:16:48.978011 17283 raft_consensus.cc:686] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca [term 1 LEADER]: Becoming Leader. State: Replica: 5f3106fa1236489f900e1f0a33cd63ca, State: Running, Role: LEADER
I20240502 08:16:48.978371 17283 consensus_queue.cc:237] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f3106fa1236489f900e1f0a33cd63ca" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 45733 } }
I20240502 08:16:48.980319 17277 ts_tablet_manager.cc:1398] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca: Time spent starting tablet: real 0.018s user 0.010s sys 0.008s
I20240502 08:16:48.980686 17289 raft_consensus.cc:686] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d [term 1 LEADER]: Becoming Leader. State: Replica: 1c87426d93224befaac733a622db504d, State: Running, Role: LEADER
I20240502 08:16:48.981053 17289 consensus_queue.cc:237] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c87426d93224befaac733a622db504d" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 35745 } }
I20240502 08:16:48.983601 16916 catalog_manager.cc:5484] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6167a8d5011843e4af9c48a0a757ad23 (127.15.81.2). New cstate: current_term: 1 leader_uuid: "6167a8d5011843e4af9c48a0a757ad23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6167a8d5011843e4af9c48a0a757ad23" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 43787 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:48.997344 16916 catalog_manager.cc:5484] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c87426d93224befaac733a622db504d (127.15.81.3). New cstate: current_term: 1 leader_uuid: "1c87426d93224befaac733a622db504d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c87426d93224befaac733a622db504d" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 35745 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:49.002825 17279 ts_tablet_manager.cc:1398] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d: Time spent starting tablet: real 0.034s user 0.000s sys 0.021s
I20240502 08:16:49.009164 16916 catalog_manager.cc:5484] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f3106fa1236489f900e1f0a33cd63ca (127.15.81.1). New cstate: current_term: 1 leader_uuid: "5f3106fa1236489f900e1f0a33cd63ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f3106fa1236489f900e1f0a33cd63ca" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 45733 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:49.033835 15684 master.cc:537] Master@127.15.81.60:42167 shutting down...
I20240502 08:16:49.040253 15684 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:49.040623 15684 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:49.040777 15684 tablet_replica.cc:330] T 00000000000000000000000000000000 P 55ab5fa0c46c459d963e41d854829123: stopping tablet replica
I20240502 08:16:49.055948 15684 master.cc:552] Master@127.15.81.60:42167 shutdown complete.
W20240502 08:16:50.028864 17226 heartbeater.cc:644] Failed to heartbeat to 127.15.81.60:42167 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.81.60:42167: connect: Connection refused (error 111)
I20240502 08:16:50.528229 17302 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 55ab5fa0c46c459d963e41d854829123)
I20240502 08:16:50.528447 17302 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } }
I20240502 08:16:50.529245 17302 leader_election.cc:288] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1603b6ada9fb44f796adc27703e340af (127.15.81.61:36075), 55ab5fa0c46c459d963e41d854829123 (127.15.81.60:42167)
I20240502 08:16:50.529799 16860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac18cf8af21c40eebbe3aad070aeffac" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "1603b6ada9fb44f796adc27703e340af" is_pre_election: true
I20240502 08:16:50.531327 17305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 55ab5fa0c46c459d963e41d854829123)
W20240502 08:16:50.531385 16771 leader_election.cc:334] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 55ab5fa0c46c459d963e41d854829123 (127.15.81.60:42167): Network error: Client connection negotiation failed: client connection to 127.15.81.60:42167: connect: Connection refused (error 111)
I20240502 08:16:50.531539 16771 leader_election.cc:302] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac18cf8af21c40eebbe3aad070aeffac; no voters: 1603b6ada9fb44f796adc27703e340af, 55ab5fa0c46c459d963e41d854829123
I20240502 08:16:50.531519 17305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } }
I20240502 08:16:50.531893 17302 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:50.532275 17305 leader_election.cc:288] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac18cf8af21c40eebbe3aad070aeffac (127.15.81.62:39313), 55ab5fa0c46c459d963e41d854829123 (127.15.81.60:42167)
I20240502 08:16:50.532639 16793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1603b6ada9fb44f796adc27703e340af" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "ac18cf8af21c40eebbe3aad070aeffac" is_pre_election: true
I20240502 08:16:50.532912 16793 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1603b6ada9fb44f796adc27703e340af in term 1.
I20240502 08:16:50.533459 16836 leader_election.cc:302] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1603b6ada9fb44f796adc27703e340af, ac18cf8af21c40eebbe3aad070aeffac; no voters:
I20240502 08:16:50.533742 17305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:16:50.533845 17305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 1 FOLLOWER]: Starting leader election (detected failure of leader 55ab5fa0c46c459d963e41d854829123)
I20240502 08:16:50.533946 17305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:16:50.534155 16836 leader_election.cc:334] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 55ab5fa0c46c459d963e41d854829123 (127.15.81.60:42167): Network error: Client connection negotiation failed: client connection to 127.15.81.60:42167: connect: Connection refused (error 111)
I20240502 08:16:50.535959 17305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } }
I20240502 08:16:50.536417 17305 leader_election.cc:288] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [CANDIDATE]: Term 2 election: Requested vote from peers ac18cf8af21c40eebbe3aad070aeffac (127.15.81.62:39313), 55ab5fa0c46c459d963e41d854829123 (127.15.81.60:42167)
I20240502 08:16:50.536873 16793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1603b6ada9fb44f796adc27703e340af" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "ac18cf8af21c40eebbe3aad070aeffac"
I20240502 08:16:50.537051 16793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:50.538817 16793 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1603b6ada9fb44f796adc27703e340af in term 2.
I20240502 08:16:50.539191 16836 leader_election.cc:302] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1603b6ada9fb44f796adc27703e340af, ac18cf8af21c40eebbe3aad070aeffac; no voters:
I20240502 08:16:50.539450 17305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:16:50.539924 17305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 2 LEADER]: Becoming Leader. State: Replica: 1603b6ada9fb44f796adc27703e340af, State: Running, Role: LEADER
W20240502 08:16:50.540254 16836 leader_election.cc:334] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 55ab5fa0c46c459d963e41d854829123 (127.15.81.60:42167): Network error: Client connection negotiation failed: client connection to 127.15.81.60:42167: connect: Connection refused (error 111)
I20240502 08:16:50.540257 17305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } }
I20240502 08:16:50.541428 17308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1603b6ada9fb44f796adc27703e340af. Latest consensus state: current_term: 2 leader_uuid: "1603b6ada9fb44f796adc27703e340af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } } }
I20240502 08:16:50.541690 17308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:50.542256 17310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:16:50.544529 17310 catalog_manager.cc:669] Loaded metadata for table test-workload [id=747d1d3e490e4d89b89e5aee06a9f3eb]
I20240502 08:16:50.546168 17310 tablet_loader.cc:96] loaded metadata for tablet 3ea63328a6ea4f3cb4836a81081d48b4 (table test-workload [id=747d1d3e490e4d89b89e5aee06a9f3eb])
I20240502 08:16:50.546502 17310 tablet_loader.cc:96] loaded metadata for tablet 40c25833f5164019a3ca32dd9b9e9a7a (table test-workload [id=747d1d3e490e4d89b89e5aee06a9f3eb])
I20240502 08:16:50.546787 17310 tablet_loader.cc:96] loaded metadata for tablet 575c5504df8e44fbac3a8c15d2d818be (table test-workload [id=747d1d3e490e4d89b89e5aee06a9f3eb])
I20240502 08:16:50.546995 17310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:16:50.547829 17310 catalog_manager.cc:1259] Loaded cluster ID: 4ce64d649e4f42a39e32cf5a121039c7
I20240502 08:16:50.547911 17310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:16:50.549010 17310 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:16:50.549782 17310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af: Loaded TSK: 0
I20240502 08:16:50.550558 17310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:16:51.061573 16793 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [term 2 FOLLOWER]: Refusing update from remote peer 1603b6ada9fb44f796adc27703e340af: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 08:16:51.062196 17314 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:16:51.063961 17315 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1603b6ada9fb44f796adc27703e340af. Latest consensus state: current_term: 2 leader_uuid: "1603b6ada9fb44f796adc27703e340af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } } }
I20240502 08:16:51.064242 17315 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:51.065688 17314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "1603b6ada9fb44f796adc27703e340af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } } }
I20240502 08:16:51.065994 17314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [sys.catalog]: This master's current role is: LEADER
W20240502 08:16:51.076279 16836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.81.60:42167: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:16:51.077793 16836 consensus_peers.cc:485] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af -> Peer 55ab5fa0c46c459d963e41d854829123 (127.15.81.60:42167): Couldn't send request to peer 55ab5fa0c46c459d963e41d854829123. Status: Network error: Client connection negotiation failed: client connection to 127.15.81.60:42167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:51.078408 17315 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "1603b6ada9fb44f796adc27703e340af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac18cf8af21c40eebbe3aad070aeffac" member_type: VOTER last_known_addr { host: "127.15.81.62" port: 39313 } } peers { permanent_uuid: "1603b6ada9fb44f796adc27703e340af" member_type: VOTER last_known_addr { host: "127.15.81.61" port: 36075 } } peers { permanent_uuid: "55ab5fa0c46c459d963e41d854829123" member_type: VOTER last_known_addr { host: "127.15.81.60" port: 42167 } } }
I20240502 08:16:51.078655 17315 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:52.017249 17070 heartbeater.cc:503] Master 127.15.81.61:36075 was elected leader, sending a full tablet report...
I20240502 08:16:52.025494 17228 heartbeater.cc:503] Master 127.15.81.61:36075 was elected leader, sending a full tablet report...
I20240502 08:16:52.034199 17149 heartbeater.cc:503] Master 127.15.81.61:36075 was elected leader, sending a full tablet report...
I20240502 08:16:52.404172 16967 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:16:52.404362 16967 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:16:52.599694 15684 tablet_server.cc:178] TabletServer@127.15.81.1:0 shutting down...
I20240502 08:16:52.608523 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:16:52.608824 15684 tablet_replica.cc:330] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca: stopping tablet replica
I20240502 08:16:52.609001 15684 raft_consensus.cc:2229] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:52.609210 15684 raft_consensus.cc:2258] T 3ea63328a6ea4f3cb4836a81081d48b4 P 5f3106fa1236489f900e1f0a33cd63ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:52.622704 15684 tablet_server.cc:194] TabletServer@127.15.81.1:0 shutdown complete.
I20240502 08:16:52.625661 15684 tablet_server.cc:178] TabletServer@127.15.81.2:0 shutting down...
I20240502 08:16:52.634357 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:16:52.634629 15684 tablet_replica.cc:330] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23: stopping tablet replica
I20240502 08:16:52.634802 15684 raft_consensus.cc:2229] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:52.634989 15684 raft_consensus.cc:2258] T 575c5504df8e44fbac3a8c15d2d818be P 6167a8d5011843e4af9c48a0a757ad23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:52.648777 15684 tablet_server.cc:194] TabletServer@127.15.81.2:0 shutdown complete.
I20240502 08:16:52.651767 15684 tablet_server.cc:178] TabletServer@127.15.81.3:0 shutting down...
I20240502 08:16:52.659696 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:16:52.659963 15684 tablet_replica.cc:330] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d: stopping tablet replica
I20240502 08:16:52.660133 15684 raft_consensus.cc:2229] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:52.660327 15684 raft_consensus.cc:2258] T 40c25833f5164019a3ca32dd9b9e9a7a P 1c87426d93224befaac733a622db504d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:52.673530 15684 tablet_server.cc:194] TabletServer@127.15.81.3:0 shutdown complete.
I20240502 08:16:52.676344 15684 master.cc:537] Master@127.15.81.62:39313 shutting down...
I20240502 08:16:52.685887 15684 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:52.686165 15684 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:52.686282 15684 tablet_replica.cc:330] T 00000000000000000000000000000000 P ac18cf8af21c40eebbe3aad070aeffac: stopping tablet replica
W20240502 08:16:52.698184 16836 consensus_peers.cc:485] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af -> Peer ac18cf8af21c40eebbe3aad070aeffac (127.15.81.62:39313): Couldn't send request to peer ac18cf8af21c40eebbe3aad070aeffac. Status: Network error: Client connection negotiation failed: client connection to 127.15.81.62:39313: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:52.699892 15684 master.cc:552] Master@127.15.81.62:39313 shutdown complete.
I20240502 08:16:52.703971 15684 master.cc:537] Master@127.15.81.61:36075 shutting down...
I20240502 08:16:52.710721 15684 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:16:52.711123 15684 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:52.711285 15684 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1603b6ada9fb44f796adc27703e340af: stopping tablet replica
I20240502 08:16:52.724938 15684 master.cc:552] Master@127.15.81.61:36075 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6530 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240502 08:16:52.745525 15684 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.81.62:35255
I20240502 08:16:52.746001 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:52.749382 17329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:52.750301 17330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:52.751029 17332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:52.751762 15684 server_base.cc:1034] running on GCE node
I20240502 08:16:52.752022 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:52.752079 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:52.752115 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637812752110 us; error 0 us; skew 500 ppm
I20240502 08:16:52.752252 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:52.760129 15684 webserver.cc:460] Webserver started at http://127.15.81.62:45427/ using document root <none> and password file <none>
I20240502 08:16:52.760351 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:52.760414 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:52.760548 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:52.761171 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/master-0-root/instance:
uuid: "0b49fccfc5174f87831da57965eacab6"
format_stamp: "Formatted at 2024-05-02 08:16:52 on dist-test-slave-rhmj"
I20240502 08:16:52.763584 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:52.765100 17337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:52.765468 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:16:52.765558 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/master-0-root
uuid: "0b49fccfc5174f87831da57965eacab6"
format_stamp: "Formatted at 2024-05-02 08:16:52 on dist-test-slave-rhmj"
I20240502 08:16:52.765645 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:52.786994 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:52.787513 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:52.799019 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.62:35255
I20240502 08:16:52.799026 17389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.62:35255 every 8 connection(s)
I20240502 08:16:52.800388 17390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.804572 17390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6: Bootstrap starting.
I20240502 08:16:52.805902 17390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.807595 17390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6: No bootstrap required, opened a new log
I20240502 08:16:52.808280 17390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b49fccfc5174f87831da57965eacab6" member_type: VOTER }
I20240502 08:16:52.808422 17390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.808485 17390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b49fccfc5174f87831da57965eacab6, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.808682 17390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b49fccfc5174f87831da57965eacab6" member_type: VOTER }
I20240502 08:16:52.808806 17390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:52.808872 17390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:52.808962 17390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.810743 17390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b49fccfc5174f87831da57965eacab6" member_type: VOTER }
I20240502 08:16:52.810956 17390 leader_election.cc:302] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b49fccfc5174f87831da57965eacab6; no voters:
I20240502 08:16:52.811352 17390 leader_election.cc:288] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:52.811460 17393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:52.811960 17393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [term 1 LEADER]: Becoming Leader. State: Replica: 0b49fccfc5174f87831da57965eacab6, State: Running, Role: LEADER
I20240502 08:16:52.812232 17393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b49fccfc5174f87831da57965eacab6" member_type: VOTER }
I20240502 08:16:52.812301 17390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:52.813042 17394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b49fccfc5174f87831da57965eacab6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b49fccfc5174f87831da57965eacab6" member_type: VOTER } }
I20240502 08:16:52.813072 17395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b49fccfc5174f87831da57965eacab6. Latest consensus state: current_term: 1 leader_uuid: "0b49fccfc5174f87831da57965eacab6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b49fccfc5174f87831da57965eacab6" member_type: VOTER } }
I20240502 08:16:52.813292 17394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:52.813382 17395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:52.814963 17398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:16:52.816216 17398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:16:52.817929 15684 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:16:52.819025 17398 catalog_manager.cc:1347] Generated new cluster ID: 3379dc8423404fae816faa857a272d62
I20240502 08:16:52.819108 17398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:16:52.845424 17398 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:16:52.846264 17398 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:16:52.854393 17398 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6: Generated new TSK 0
I20240502 08:16:52.854635 17398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:16:52.882889 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:52.886557 17411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:52.887255 17412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:52.887955 17414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:52.888911 15684 server_base.cc:1034] running on GCE node
I20240502 08:16:52.889175 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:52.889240 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:52.889281 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637812889278 us; error 0 us; skew 500 ppm
I20240502 08:16:52.889422 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:52.890599 15684 webserver.cc:460] Webserver started at http://127.15.81.1:44671/ using document root <none> and password file <none>
I20240502 08:16:52.890792 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:52.890856 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:52.890977 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:52.891644 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-0-root/instance:
uuid: "ffc96e1a70b346a2aafc7d8f350236f9"
format_stamp: "Formatted at 2024-05-02 08:16:52 on dist-test-slave-rhmj"
I20240502 08:16:52.894071 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:52.895469 17420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:52.895852 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:16:52.895941 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-0-root
uuid: "ffc96e1a70b346a2aafc7d8f350236f9"
format_stamp: "Formatted at 2024-05-02 08:16:52 on dist-test-slave-rhmj"
I20240502 08:16:52.896032 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:52.904592 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:52.905104 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:52.905625 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:52.906494 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:52.906546 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:52.906610 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:52.906653 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:52.920081 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.1:37511
I20240502 08:16:52.920096 17483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.1:37511 every 8 connection(s)
I20240502 08:16:52.921733 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:52.925374 17488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:52.926422 17489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:52.927213 17491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:52.927871 15684 server_base.cc:1034] running on GCE node
I20240502 08:16:52.928121 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:52.928179 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:52.928128 17484 heartbeater.cc:347] Connected to a master server at 127.15.81.62:35255
I20240502 08:16:52.928205 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637812928201 us; error 0 us; skew 500 ppm
I20240502 08:16:52.928279 17484 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:52.928344 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:52.928594 17484 heartbeater.cc:511] Master 127.15.81.62:35255 requested a full tablet report, sending...
I20240502 08:16:52.929433 17354 ts_manager.cc:194] Registered new tserver with Master: ffc96e1a70b346a2aafc7d8f350236f9 (127.15.81.1:37511)
I20240502 08:16:52.929664 15684 webserver.cc:460] Webserver started at http://127.15.81.2:45043/ using document root <none> and password file <none>
I20240502 08:16:52.929864 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:52.929919 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:52.930025 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:52.930601 17354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34312
I20240502 08:16:52.930655 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-1-root/instance:
uuid: "7f7e54ce562847258886001ba89ba1d1"
format_stamp: "Formatted at 2024-05-02 08:16:52 on dist-test-slave-rhmj"
I20240502 08:16:52.933046 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 08:16:52.934604 17496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:52.934907 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240502 08:16:52.935001 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-1-root
uuid: "7f7e54ce562847258886001ba89ba1d1"
format_stamp: "Formatted at 2024-05-02 08:16:52 on dist-test-slave-rhmj"
I20240502 08:16:52.935096 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:52.956892 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:52.957443 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:52.957978 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:52.958750 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:52.958814 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:52.958904 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:52.958938 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:52.972404 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.2:44387
I20240502 08:16:52.972443 17559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.2:44387 every 8 connection(s)
I20240502 08:16:52.973937 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:52.977614 17564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:52.978670 17565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:52.979346 17567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:52.980113 15684 server_base.cc:1034] running on GCE node
I20240502 08:16:52.980131 17560 heartbeater.cc:347] Connected to a master server at 127.15.81.62:35255
I20240502 08:16:52.980301 17560 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:52.980414 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:16:52.980470 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:16:52.980505 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637812980501 us; error 0 us; skew 500 ppm
I20240502 08:16:52.980623 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:52.980664 17560 heartbeater.cc:511] Master 127.15.81.62:35255 requested a full tablet report, sending...
I20240502 08:16:52.981606 17354 ts_manager.cc:194] Registered new tserver with Master: 7f7e54ce562847258886001ba89ba1d1 (127.15.81.2:44387)
I20240502 08:16:52.982167 15684 webserver.cc:460] Webserver started at http://127.15.81.3:33853/ using document root <none> and password file <none>
I20240502 08:16:52.982323 17354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34328
I20240502 08:16:52.982374 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:52.982465 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:52.982606 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:52.983321 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-2-root/instance:
uuid: "4f232f5620d546f997857c6455ca72ab"
format_stamp: "Formatted at 2024-05-02 08:16:52 on dist-test-slave-rhmj"
I20240502 08:16:52.985879 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:52.987470 17572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:52.987833 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:16:52.987952 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-2-root
uuid: "4f232f5620d546f997857c6455ca72ab"
format_stamp: "Formatted at 2024-05-02 08:16:52 on dist-test-slave-rhmj"
I20240502 08:16:52.988058 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:53.008729 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:53.009330 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:53.009874 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:53.010684 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:53.010735 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:53.010807 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:53.010851 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:53.025090 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.3:45321
I20240502 08:16:53.025118 17635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.3:45321 every 8 connection(s)
I20240502 08:16:53.031462 17636 heartbeater.cc:347] Connected to a master server at 127.15.81.62:35255
I20240502 08:16:53.031587 17636 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:53.031939 17636 heartbeater.cc:511] Master 127.15.81.62:35255 requested a full tablet report, sending...
I20240502 08:16:53.032694 17354 ts_manager.cc:194] Registered new tserver with Master: 4f232f5620d546f997857c6455ca72ab (127.15.81.3:45321)
I20240502 08:16:53.032802 15684 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006775856s
I20240502 08:16:53.033452 17354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34344
I20240502 08:16:53.931878 17484 heartbeater.cc:503] Master 127.15.81.62:35255 was elected leader, sending a full tablet report...
I20240502 08:16:53.983641 17560 heartbeater.cc:503] Master 127.15.81.62:35255 was elected leader, sending a full tablet report...
I20240502 08:16:54.034772 17636 heartbeater.cc:503] Master 127.15.81.62:35255 was elected leader, sending a full tablet report...
I20240502 08:16:54.060292 15684 test_util.cc:268] Using random seed: 970151162
I20240502 08:16:54.074079 17354 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34354:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.074743 17354 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.090987 17448 tablet_service.cc:1467] Processing CreateTablet for tablet 5863fbeaebcb43229cf2e665a9a95ca1 (DEFAULT_TABLE table=test-workload [id=a4587e185e714bcb9a04b58a0125e0ea]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 08:16:54.091638 17448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5863fbeaebcb43229cf2e665a9a95ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.093403 17524 tablet_service.cc:1467] Processing CreateTablet for tablet 12c6fa3938c249489af92bb44a822999 (DEFAULT_TABLE table=test-workload [id=a4587e185e714bcb9a04b58a0125e0ea]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 08:16:54.093428 17523 tablet_service.cc:1467] Processing CreateTablet for tablet 5863fbeaebcb43229cf2e665a9a95ca1 (DEFAULT_TABLE table=test-workload [id=a4587e185e714bcb9a04b58a0125e0ea]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 08:16:54.093889 17524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c6fa3938c249489af92bb44a822999. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.093802 17600 tablet_service.cc:1467] Processing CreateTablet for tablet 5863fbeaebcb43229cf2e665a9a95ca1 (DEFAULT_TABLE table=test-workload [id=a4587e185e714bcb9a04b58a0125e0ea]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 08:16:54.094132 17523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5863fbeaebcb43229cf2e665a9a95ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.094235 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5863fbeaebcb43229cf2e665a9a95ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.096179 17601 tablet_service.cc:1467] Processing CreateTablet for tablet 12c6fa3938c249489af92bb44a822999 (DEFAULT_TABLE table=test-workload [id=a4587e185e714bcb9a04b58a0125e0ea]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 08:16:54.096607 17601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c6fa3938c249489af92bb44a822999. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.090987 17449 tablet_service.cc:1467] Processing CreateTablet for tablet 12c6fa3938c249489af92bb44a822999 (DEFAULT_TABLE table=test-workload [id=a4587e185e714bcb9a04b58a0125e0ea]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 08:16:54.100170 17449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c6fa3938c249489af92bb44a822999. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.111979 17658 tablet_bootstrap.cc:492] T 12c6fa3938c249489af92bb44a822999 P ffc96e1a70b346a2aafc7d8f350236f9: Bootstrap starting.
I20240502 08:16:54.112154 17656 tablet_bootstrap.cc:492] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab: Bootstrap starting.
I20240502 08:16:54.113461 17657 tablet_bootstrap.cc:492] T 12c6fa3938c249489af92bb44a822999 P 7f7e54ce562847258886001ba89ba1d1: Bootstrap starting.
I20240502 08:16:54.114089 17658 tablet_bootstrap.cc:654] T 12c6fa3938c249489af92bb44a822999 P ffc96e1a70b346a2aafc7d8f350236f9: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.114128 17656 tablet_bootstrap.cc:654] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.115914 17657 tablet_bootstrap.cc:654] T 12c6fa3938c249489af92bb44a822999 P 7f7e54ce562847258886001ba89ba1d1: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.117637 17657 tablet_bootstrap.cc:492] T 12c6fa3938c249489af92bb44a822999 P 7f7e54ce562847258886001ba89ba1d1: No bootstrap required, opened a new log
I20240502 08:16:54.117789 17657 ts_tablet_manager.cc:1367] T 12c6fa3938c249489af92bb44a822999 P 7f7e54ce562847258886001ba89ba1d1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:54.118395 17657 raft_consensus.cc:348] T 12c6fa3938c249489af92bb44a822999 P 7f7e54ce562847258886001ba89ba1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } }
I20240502 08:16:54.118572 17657 raft_consensus.cc:374] T 12c6fa3938c249489af92bb44a822999 P 7f7e54ce562847258886001ba89ba1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.118695 17657 raft_consensus.cc:729] T 12c6fa3938c249489af92bb44a822999 P 7f7e54ce562847258886001ba89ba1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7e54ce562847258886001ba89ba1d1, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.118927 17657 consensus_queue.cc:260] T 12c6fa3938c249489af92bb44a822999 P 7f7e54ce562847258886001ba89ba1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } }
I20240502 08:16:54.119678 17657 ts_tablet_manager.cc:1398] T 12c6fa3938c249489af92bb44a822999 P 7f7e54ce562847258886001ba89ba1d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:54.120241 17657 tablet_bootstrap.cc:492] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1: Bootstrap starting.
I20240502 08:16:54.121906 17657 tablet_bootstrap.cc:654] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.122996 17656 tablet_bootstrap.cc:492] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab: No bootstrap required, opened a new log
I20240502 08:16:54.123008 17658 tablet_bootstrap.cc:492] T 12c6fa3938c249489af92bb44a822999 P ffc96e1a70b346a2aafc7d8f350236f9: No bootstrap required, opened a new log
I20240502 08:16:54.123212 17656 ts_tablet_manager.cc:1367] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 08:16:54.123268 17658 ts_tablet_manager.cc:1367] T 12c6fa3938c249489af92bb44a822999 P ffc96e1a70b346a2aafc7d8f350236f9: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20240502 08:16:54.124132 17658 raft_consensus.cc:348] T 12c6fa3938c249489af92bb44a822999 P ffc96e1a70b346a2aafc7d8f350236f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } }
I20240502 08:16:54.124372 17658 raft_consensus.cc:374] T 12c6fa3938c249489af92bb44a822999 P ffc96e1a70b346a2aafc7d8f350236f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.124485 17658 raft_consensus.cc:729] T 12c6fa3938c249489af92bb44a822999 P ffc96e1a70b346a2aafc7d8f350236f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc96e1a70b346a2aafc7d8f350236f9, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.124754 17658 consensus_queue.cc:260] T 12c6fa3938c249489af92bb44a822999 P ffc96e1a70b346a2aafc7d8f350236f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } }
I20240502 08:16:54.125113 17656 raft_consensus.cc:348] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } }
I20240502 08:16:54.125356 17656 raft_consensus.cc:374] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.125468 17656 raft_consensus.cc:729] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f232f5620d546f997857c6455ca72ab, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.125679 17656 consensus_queue.cc:260] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } }
I20240502 08:16:54.126354 17658 ts_tablet_manager.cc:1398] T 12c6fa3938c249489af92bb44a822999 P ffc96e1a70b346a2aafc7d8f350236f9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:54.126668 17658 tablet_bootstrap.cc:492] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9: Bootstrap starting.
I20240502 08:16:54.128365 17658 tablet_bootstrap.cc:654] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.129676 17656 ts_tablet_manager.cc:1398] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:16:54.129998 17656 tablet_bootstrap.cc:492] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab: Bootstrap starting.
I20240502 08:16:54.130038 17657 tablet_bootstrap.cc:492] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1: No bootstrap required, opened a new log
I20240502 08:16:54.130201 17657 ts_tablet_manager.cc:1367] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240502 08:16:54.131013 17657 raft_consensus.cc:348] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } }
I20240502 08:16:54.131235 17657 raft_consensus.cc:374] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.131328 17657 raft_consensus.cc:729] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7e54ce562847258886001ba89ba1d1, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.131687 17656 tablet_bootstrap.cc:654] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.131589 17657 consensus_queue.cc:260] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } }
I20240502 08:16:54.132125 17657 ts_tablet_manager.cc:1398] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:54.133332 17658 tablet_bootstrap.cc:492] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9: No bootstrap required, opened a new log
I20240502 08:16:54.133491 17658 ts_tablet_manager.cc:1367] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20240502 08:16:54.133780 17656 tablet_bootstrap.cc:492] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab: No bootstrap required, opened a new log
I20240502 08:16:54.133937 17656 ts_tablet_manager.cc:1367] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 08:16:54.134333 17658 raft_consensus.cc:348] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } }
I20240502 08:16:54.134474 17656 raft_consensus.cc:348] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } }
I20240502 08:16:54.134825 17656 raft_consensus.cc:374] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.134644 17658 raft_consensus.cc:374] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.134994 17656 raft_consensus.cc:729] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f232f5620d546f997857c6455ca72ab, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.135088 17658 raft_consensus.cc:729] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc96e1a70b346a2aafc7d8f350236f9, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.135363 17656 consensus_queue.cc:260] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } }
I20240502 08:16:54.135450 17658 consensus_queue.cc:260] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } }
I20240502 08:16:54.136022 17658 ts_tablet_manager.cc:1398] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:54.136189 17656 ts_tablet_manager.cc:1398] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:54.190719 17665 raft_consensus.cc:482] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.190912 17665 raft_consensus.cc:504] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } }
I20240502 08:16:54.191800 17665 leader_election.cc:288] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffc96e1a70b346a2aafc7d8f350236f9 (127.15.81.1:37511), 7f7e54ce562847258886001ba89ba1d1 (127.15.81.2:44387)
I20240502 08:16:54.199649 17534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c6fa3938c249489af92bb44a822999" candidate_uuid: "4f232f5620d546f997857c6455ca72ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f7e54ce562847258886001ba89ba1d1" is_pre_election: true
I20240502 08:16:54.199695 17459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c6fa3938c249489af92bb44a822999" candidate_uuid: "4f232f5620d546f997857c6455ca72ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" is_pre_election: true
I20240502 08:16:54.199936 17534 raft_consensus.cc:2454] T 12c6fa3938c249489af92bb44a822999 P 7f7e54ce562847258886001ba89ba1d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f232f5620d546f997857c6455ca72ab in term 0.
I20240502 08:16:54.199965 17459 raft_consensus.cc:2454] T 12c6fa3938c249489af92bb44a822999 P ffc96e1a70b346a2aafc7d8f350236f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f232f5620d546f997857c6455ca72ab in term 0.
I20240502 08:16:54.200426 17574 leader_election.cc:302] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4f232f5620d546f997857c6455ca72ab, 7f7e54ce562847258886001ba89ba1d1, ffc96e1a70b346a2aafc7d8f350236f9; no voters:
I20240502 08:16:54.200734 17665 raft_consensus.cc:2781] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:54.200856 17665 raft_consensus.cc:482] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:54.200932 17665 raft_consensus.cc:3037] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.202787 17665 raft_consensus.cc:504] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } }
I20240502 08:16:54.203296 17665 leader_election.cc:288] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [CANDIDATE]: Term 1 election: Requested vote from peers ffc96e1a70b346a2aafc7d8f350236f9 (127.15.81.1:37511), 7f7e54ce562847258886001ba89ba1d1 (127.15.81.2:44387)
I20240502 08:16:54.203753 17459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c6fa3938c249489af92bb44a822999" candidate_uuid: "4f232f5620d546f997857c6455ca72ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc96e1a70b346a2aafc7d8f350236f9"
I20240502 08:16:54.203965 17459 raft_consensus.cc:3037] T 12c6fa3938c249489af92bb44a822999 P ffc96e1a70b346a2aafc7d8f350236f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.203828 17534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c6fa3938c249489af92bb44a822999" candidate_uuid: "4f232f5620d546f997857c6455ca72ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f7e54ce562847258886001ba89ba1d1"
I20240502 08:16:54.204216 17534 raft_consensus.cc:3037] T 12c6fa3938c249489af92bb44a822999 P 7f7e54ce562847258886001ba89ba1d1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.206012 17459 raft_consensus.cc:2454] T 12c6fa3938c249489af92bb44a822999 P ffc96e1a70b346a2aafc7d8f350236f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f232f5620d546f997857c6455ca72ab in term 1.
I20240502 08:16:54.206403 17573 leader_election.cc:302] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f232f5620d546f997857c6455ca72ab, ffc96e1a70b346a2aafc7d8f350236f9; no voters:
I20240502 08:16:54.206533 17534 raft_consensus.cc:2454] T 12c6fa3938c249489af92bb44a822999 P 7f7e54ce562847258886001ba89ba1d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f232f5620d546f997857c6455ca72ab in term 1.
I20240502 08:16:54.206704 17665 raft_consensus.cc:2781] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.207304 17665 raft_consensus.cc:686] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [term 1 LEADER]: Becoming Leader. State: Replica: 4f232f5620d546f997857c6455ca72ab, State: Running, Role: LEADER
I20240502 08:16:54.207666 17665 consensus_queue.cc:237] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } }
I20240502 08:16:54.210038 17353 catalog_manager.cc:5484] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f232f5620d546f997857c6455ca72ab (127.15.81.3). New cstate: current_term: 1 leader_uuid: "4f232f5620d546f997857c6455ca72ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:54.344373 17664 raft_consensus.cc:482] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.344611 17664 raft_consensus.cc:504] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } }
I20240502 08:16:54.345588 17664 leader_election.cc:288] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f7e54ce562847258886001ba89ba1d1 (127.15.81.2:44387), 4f232f5620d546f997857c6455ca72ab (127.15.81.3:45321)
I20240502 08:16:54.351579 17534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5863fbeaebcb43229cf2e665a9a95ca1" candidate_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f7e54ce562847258886001ba89ba1d1" is_pre_election: true
I20240502 08:16:54.351584 17611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5863fbeaebcb43229cf2e665a9a95ca1" candidate_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f232f5620d546f997857c6455ca72ab" is_pre_election: true
I20240502 08:16:54.351928 17611 raft_consensus.cc:2454] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffc96e1a70b346a2aafc7d8f350236f9 in term 0.
I20240502 08:16:54.351935 17534 raft_consensus.cc:2454] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffc96e1a70b346a2aafc7d8f350236f9 in term 0.
I20240502 08:16:54.352365 17424 leader_election.cc:302] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f232f5620d546f997857c6455ca72ab, ffc96e1a70b346a2aafc7d8f350236f9; no voters:
I20240502 08:16:54.352612 17664 raft_consensus.cc:2781] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:54.352748 17664 raft_consensus.cc:482] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:54.352833 17664 raft_consensus.cc:3037] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.354962 17664 raft_consensus.cc:504] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } }
I20240502 08:16:54.355464 17664 leader_election.cc:288] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [CANDIDATE]: Term 1 election: Requested vote from peers 7f7e54ce562847258886001ba89ba1d1 (127.15.81.2:44387), 4f232f5620d546f997857c6455ca72ab (127.15.81.3:45321)
I20240502 08:16:54.355966 17611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5863fbeaebcb43229cf2e665a9a95ca1" candidate_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f232f5620d546f997857c6455ca72ab"
I20240502 08:16:54.355965 17534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5863fbeaebcb43229cf2e665a9a95ca1" candidate_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f7e54ce562847258886001ba89ba1d1"
I20240502 08:16:54.356199 17611 raft_consensus.cc:3037] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.356215 17534 raft_consensus.cc:3037] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.358947 17611 raft_consensus.cc:2454] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffc96e1a70b346a2aafc7d8f350236f9 in term 1.
I20240502 08:16:54.358942 17534 raft_consensus.cc:2454] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffc96e1a70b346a2aafc7d8f350236f9 in term 1.
I20240502 08:16:54.359364 17422 leader_election.cc:302] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f7e54ce562847258886001ba89ba1d1, ffc96e1a70b346a2aafc7d8f350236f9; no voters:
I20240502 08:16:54.359616 17664 raft_consensus.cc:2781] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.360030 17664 raft_consensus.cc:686] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [term 1 LEADER]: Becoming Leader. State: Replica: ffc96e1a70b346a2aafc7d8f350236f9, State: Running, Role: LEADER
I20240502 08:16:54.360278 17664 consensus_queue.cc:237] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } }
I20240502 08:16:54.362746 17353 catalog_manager.cc:5484] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffc96e1a70b346a2aafc7d8f350236f9 (127.15.81.1). New cstate: current_term: 1 leader_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:54.703907 17665 consensus_queue.cc:1035] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:54.709654 17670 consensus_queue.cc:1035] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:54.816622 17664 consensus_queue.cc:1035] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:54.817860 17408 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:16:54.818693 17408 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:16:54.818879 17408 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:16:54.822433 17674 consensus_queue.cc:1035] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:56.819350 17408 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:16:56.820194 17408 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:16:56.820354 17408 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:16:58.820927 17408 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:16:58.821585 17408 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:16:58.821738 17408 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:00.822242 17408 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:00.822968 17408 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:00.823104 17408 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:02.823690 17408 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:02.824401 17408 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:02.824581 17408 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:04.825062 17408 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:04.825778 17408 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:04.825925 17408 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:05.434044 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:05.438339 17691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:05.438942 17690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:05.439815 17693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:05.440773 15684 server_base.cc:1034] running on GCE node
I20240502 08:17:05.441205 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:17:05.441293 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:17:05.441336 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637825441323 us; error 0 us; skew 500 ppm
I20240502 08:17:05.441474 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:05.447678 15684 webserver.cc:460] Webserver started at http://127.15.81.4:36065/ using document root <none> and password file <none>
I20240502 08:17:05.447932 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:05.448019 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:05.448149 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:05.448827 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-3-root/instance:
uuid: "5b0b8876d3c34e869038adc820e5a648"
format_stamp: "Formatted at 2024-05-02 08:17:05 on dist-test-slave-rhmj"
I20240502 08:17:05.451566 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:17:05.453362 17698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:05.453814 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 08:17:05.453933 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-3-root
uuid: "5b0b8876d3c34e869038adc820e5a648"
format_stamp: "Formatted at 2024-05-02 08:17:05 on dist-test-slave-rhmj"
I20240502 08:17:05.454078 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:05.465569 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:05.466207 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:05.466953 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:05.467945 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:05.468047 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:05.468166 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:05.468240 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:05.483912 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.4:35789
I20240502 08:17:05.483952 17760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.4:35789 every 8 connection(s)
I20240502 08:17:05.491149 17761 heartbeater.cc:347] Connected to a master server at 127.15.81.62:35255
I20240502 08:17:05.491295 17761 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:05.491724 17761 heartbeater.cc:511] Master 127.15.81.62:35255 requested a full tablet report, sending...
I20240502 08:17:05.492683 17353 ts_manager.cc:194] Registered new tserver with Master: 5b0b8876d3c34e869038adc820e5a648 (127.15.81.4:35789)
I20240502 08:17:05.493482 17353 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36398
I20240502 08:17:05.841281 17459 consensus_queue.cc:237] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } attrs { replace: true } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: true } }
I20240502 08:17:05.843957 17534 raft_consensus.cc:1261] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1 [term 1 FOLLOWER]: Refusing update from remote peer ffc96e1a70b346a2aafc7d8f350236f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:05.844013 17611 raft_consensus.cc:1261] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab [term 1 FOLLOWER]: Refusing update from remote peer ffc96e1a70b346a2aafc7d8f350236f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:05.844727 17768 consensus_queue.cc:1035] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:17:05.845072 17689 consensus_queue.cc:1035] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:17:05.847904 17689 raft_consensus.cc:2932] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5b0b8876d3c34e869038adc820e5a648 (127.15.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } attrs { replace: true } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: true } } }
I20240502 08:17:05.850361 17611 raft_consensus.cc:2932] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5b0b8876d3c34e869038adc820e5a648 (127.15.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } attrs { replace: true } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: true } } }
I20240502 08:17:05.852532 17533 raft_consensus.cc:2932] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5b0b8876d3c34e869038adc820e5a648 (127.15.81.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } attrs { replace: true } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: true } } }
I20240502 08:17:05.856266 17353 catalog_manager.cc:5484] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab reported cstate change: config changed from index -1 to 2, NON_VOTER 5b0b8876d3c34e869038adc820e5a648 (127.15.81.4) added. New cstate: current_term: 1 leader_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } attrs { replace: true } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: true } } }
W20240502 08:17:05.857615 17423 consensus_peers.cc:485] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 -> Peer 5b0b8876d3c34e869038adc820e5a648 (127.15.81.4:35789): Couldn't send request to peer 5b0b8876d3c34e869038adc820e5a648. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5863fbeaebcb43229cf2e665a9a95ca1. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:06.381078 17780 ts_tablet_manager.cc:902] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648: Initiating tablet copy from peer ffc96e1a70b346a2aafc7d8f350236f9 (127.15.81.1:37511)
I20240502 08:17:06.382478 17780 tablet_copy_client.cc:308] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648: tablet copy: Beginning tablet copy session from remote peer at address 127.15.81.1:37511
I20240502 08:17:06.389153 17469 tablet_copy_service.cc:140] P ffc96e1a70b346a2aafc7d8f350236f9: Received BeginTabletCopySession request for tablet 5863fbeaebcb43229cf2e665a9a95ca1 from peer 5b0b8876d3c34e869038adc820e5a648 ({username='slave'} at 127.0.0.1:57880)
I20240502 08:17:06.389298 17469 tablet_copy_service.cc:161] P ffc96e1a70b346a2aafc7d8f350236f9: Beginning new tablet copy session on tablet 5863fbeaebcb43229cf2e665a9a95ca1 from peer 5b0b8876d3c34e869038adc820e5a648 at {username='slave'} at 127.0.0.1:57880: session id = 5b0b8876d3c34e869038adc820e5a648-5863fbeaebcb43229cf2e665a9a95ca1
I20240502 08:17:06.391381 17469 tablet_copy_source_session.cc:205] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:06.392575 17780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5863fbeaebcb43229cf2e665a9a95ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.397779 17780 tablet_copy_client.cc:783] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:06.398108 17780 tablet_copy_client.cc:647] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:06.399842 17780 tablet_copy_client.cc:523] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:06.402377 17780 tablet_bootstrap.cc:492] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648: Bootstrap starting.
I20240502 08:17:06.407879 17780 tablet_bootstrap.cc:492] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:06.408221 17780 tablet_bootstrap.cc:492] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648: Bootstrap complete.
I20240502 08:17:06.408457 17780 ts_tablet_manager.cc:1367] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648: Time spent bootstrapping tablet: real 0.006s user 0.007s sys 0.000s
I20240502 08:17:06.409109 17780 raft_consensus.cc:348] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } attrs { replace: true } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: true } }
I20240502 08:17:06.409323 17780 raft_consensus.cc:729] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b0b8876d3c34e869038adc820e5a648, State: Initialized, Role: LEARNER
I20240502 08:17:06.409530 17780 consensus_queue.cc:260] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } attrs { replace: true } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: true } }
I20240502 08:17:06.410410 17780 ts_tablet_manager.cc:1398] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:06.410470 17761 heartbeater.cc:503] Master 127.15.81.62:35255 was elected leader, sending a full tablet report...
I20240502 08:17:06.411171 17469 tablet_copy_service.cc:342] P ffc96e1a70b346a2aafc7d8f350236f9: Request end of tablet copy session 5b0b8876d3c34e869038adc820e5a648-5863fbeaebcb43229cf2e665a9a95ca1 received from {username='slave'} at 127.0.0.1:57880
I20240502 08:17:06.411284 17469 tablet_copy_service.cc:433] P ffc96e1a70b346a2aafc7d8f350236f9: ending tablet copy session 5b0b8876d3c34e869038adc820e5a648-5863fbeaebcb43229cf2e665a9a95ca1 on tablet 5863fbeaebcb43229cf2e665a9a95ca1 with peer 5b0b8876d3c34e869038adc820e5a648
I20240502 08:17:06.826366 17408 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:06.827230 17408 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:06.827361 17408 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:06.941447 17736 raft_consensus.cc:1203] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:17:07.529302 17791 raft_consensus.cc:1050] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9: attempting to promote NON_VOTER 5b0b8876d3c34e869038adc820e5a648 to VOTER
I20240502 08:17:07.530369 17791 consensus_queue.cc:237] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } attrs { replace: true } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: false } }
I20240502 08:17:07.532655 17611 raft_consensus.cc:1261] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab [term 1 FOLLOWER]: Refusing update from remote peer ffc96e1a70b346a2aafc7d8f350236f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:07.532665 17736 raft_consensus.cc:1261] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648 [term 1 LEARNER]: Refusing update from remote peer ffc96e1a70b346a2aafc7d8f350236f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:07.533177 17791 consensus_queue.cc:1035] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:17:07.533582 17533 raft_consensus.cc:1261] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1 [term 1 FOLLOWER]: Refusing update from remote peer ffc96e1a70b346a2aafc7d8f350236f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:07.533491 17793 consensus_queue.cc:1035] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:17:07.534312 17793 consensus_queue.cc:1035] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:17:07.538091 17789 raft_consensus.cc:2932] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b0b8876d3c34e869038adc820e5a648 (127.15.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } attrs { replace: true } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: false } } }
I20240502 08:17:07.539007 17533 raft_consensus.cc:2932] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b0b8876d3c34e869038adc820e5a648 (127.15.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } attrs { replace: true } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: false } } }
I20240502 08:17:07.541555 17736 raft_consensus.cc:2932] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b0b8876d3c34e869038adc820e5a648 (127.15.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } attrs { replace: true } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: false } } }
I20240502 08:17:07.542712 17611 raft_consensus.cc:2932] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b0b8876d3c34e869038adc820e5a648 (127.15.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } attrs { replace: true } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: false } } }
I20240502 08:17:07.544492 17354 catalog_manager.cc:5484] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 reported cstate change: config changed from index 2 to 3, 5b0b8876d3c34e869038adc820e5a648 (127.15.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f232f5620d546f997857c6455ca72ab" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 45321 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:07.552837 17459 consensus_queue.cc:237] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: false } }
I20240502 08:17:07.554440 17736 raft_consensus.cc:1261] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648 [term 1 FOLLOWER]: Refusing update from remote peer ffc96e1a70b346a2aafc7d8f350236f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:07.554598 17533 raft_consensus.cc:1261] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1 [term 1 FOLLOWER]: Refusing update from remote peer ffc96e1a70b346a2aafc7d8f350236f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:07.555090 17791 consensus_queue.cc:1035] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:07.555637 17789 consensus_queue.cc:1035] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:07.557232 17791 raft_consensus.cc:2932] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4f232f5620d546f997857c6455ca72ab (127.15.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: false } } }
I20240502 08:17:07.558104 17736 raft_consensus.cc:2932] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4f232f5620d546f997857c6455ca72ab (127.15.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: false } } }
I20240502 08:17:07.558514 17533 raft_consensus.cc:2932] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4f232f5620d546f997857c6455ca72ab (127.15.81.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: false } } }
I20240502 08:17:07.561177 17338 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5863fbeaebcb43229cf2e665a9a95ca1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:07.562530 17352 catalog_manager.cc:5484] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 reported cstate change: config changed from index 3 to 4, VOTER 4f232f5620d546f997857c6455ca72ab (127.15.81.3) evicted. New cstate: current_term: 1 leader_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ffc96e1a70b346a2aafc7d8f350236f9" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f7e54ce562847258886001ba89ba1d1" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 44387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b0b8876d3c34e869038adc820e5a648" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:07.566597 17600 tablet_service.cc:1514] Processing DeleteTablet for tablet 5863fbeaebcb43229cf2e665a9a95ca1 with delete_type TABLET_DATA_TOMBSTONED (TS 4f232f5620d546f997857c6455ca72ab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48748
I20240502 08:17:07.567653 17803 tablet_replica.cc:330] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab: stopping tablet replica
I20240502 08:17:07.567850 17803 raft_consensus.cc:2229] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:07.568117 17803 raft_consensus.cc:2258] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:07.568894 17803 ts_tablet_manager.cc:1875] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:07.573299 17803 ts_tablet_manager.cc:1888] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:17:07.573418 17803 log.cc:1196] T 5863fbeaebcb43229cf2e665a9a95ca1 P 4f232f5620d546f997857c6455ca72ab: Deleting WAL directory at /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714637795017386-15684-0/minicluster-data/ts-2-root/wals/5863fbeaebcb43229cf2e665a9a95ca1
I20240502 08:17:07.574137 17341 catalog_manager.cc:4830] TS 4f232f5620d546f997857c6455ca72ab (127.15.81.3:45321): tablet 5863fbeaebcb43229cf2e665a9a95ca1 (table test-workload [id=a4587e185e714bcb9a04b58a0125e0ea]) successfully deleted
I20240502 08:17:08.827898 17408 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:08.828553 17408 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:08.828686 17408 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:10.829185 17408 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:10.829895 17408 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:10.830034 17408 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:12.830564 17408 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:12.831305 17408 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:12.831434 17408 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:14.831936 17408 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:14.832613 17408 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:14.832743 17408 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:16.833328 17408 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:16.834149 17408 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:16.834293 17408 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:18.834682 17408 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:18.835410 17408 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:18.835546 17408 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:19.567512 15684 tablet_server.cc:178] TabletServer@127.15.81.1:0 shutting down...
I20240502 08:17:19.578619 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:17:19.578907 15684 tablet_replica.cc:330] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9: stopping tablet replica
I20240502 08:17:19.579138 15684 raft_consensus.cc:2229] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:19.579450 15684 raft_consensus.cc:2258] T 5863fbeaebcb43229cf2e665a9a95ca1 P ffc96e1a70b346a2aafc7d8f350236f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:19.580070 15684 tablet_replica.cc:330] T 12c6fa3938c249489af92bb44a822999 P ffc96e1a70b346a2aafc7d8f350236f9: stopping tablet replica
I20240502 08:17:19.580221 15684 raft_consensus.cc:2229] T 12c6fa3938c249489af92bb44a822999 P ffc96e1a70b346a2aafc7d8f350236f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:19.580371 15684 raft_consensus.cc:2258] T 12c6fa3938c249489af92bb44a822999 P ffc96e1a70b346a2aafc7d8f350236f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:19.597321 15684 tablet_server.cc:194] TabletServer@127.15.81.1:0 shutdown complete.
I20240502 08:17:19.600562 15684 tablet_server.cc:178] TabletServer@127.15.81.2:0 shutting down...
I20240502 08:17:19.609066 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:17:19.609337 15684 tablet_replica.cc:330] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1: stopping tablet replica
I20240502 08:17:19.609524 15684 raft_consensus.cc:2229] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:19.609711 15684 raft_consensus.cc:2258] T 5863fbeaebcb43229cf2e665a9a95ca1 P 7f7e54ce562847258886001ba89ba1d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:19.610258 15684 tablet_replica.cc:330] T 12c6fa3938c249489af92bb44a822999 P 7f7e54ce562847258886001ba89ba1d1: stopping tablet replica
I20240502 08:17:19.610375 15684 raft_consensus.cc:2229] T 12c6fa3938c249489af92bb44a822999 P 7f7e54ce562847258886001ba89ba1d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:19.610488 15684 raft_consensus.cc:2258] T 12c6fa3938c249489af92bb44a822999 P 7f7e54ce562847258886001ba89ba1d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:19.624473 15684 tablet_server.cc:194] TabletServer@127.15.81.2:0 shutdown complete.
I20240502 08:17:19.627861 15684 tablet_server.cc:178] TabletServer@127.15.81.3:0 shutting down...
I20240502 08:17:19.641148 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:17:19.641424 15684 tablet_replica.cc:330] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab: stopping tablet replica
I20240502 08:17:19.641625 15684 raft_consensus.cc:2229] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:19.642047 15684 raft_consensus.cc:2258] T 12c6fa3938c249489af92bb44a822999 P 4f232f5620d546f997857c6455ca72ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:19.656458 15684 tablet_server.cc:194] TabletServer@127.15.81.3:0 shutdown complete.
I20240502 08:17:19.660029 15684 tablet_server.cc:178] TabletServer@127.15.81.4:0 shutting down...
I20240502 08:17:19.669032 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:17:19.669294 15684 tablet_replica.cc:330] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648: stopping tablet replica
I20240502 08:17:19.669492 15684 raft_consensus.cc:2229] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:19.669713 15684 raft_consensus.cc:2258] T 5863fbeaebcb43229cf2e665a9a95ca1 P 5b0b8876d3c34e869038adc820e5a648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:19.683166 15684 tablet_server.cc:194] TabletServer@127.15.81.4:0 shutdown complete.
I20240502 08:17:19.686249 15684 master.cc:537] Master@127.15.81.62:35255 shutting down...
I20240502 08:17:19.692801 15684 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:19.693022 15684 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:19.693104 15684 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0b49fccfc5174f87831da57965eacab6: stopping tablet replica
I20240502 08:17:19.706470 15684 master.cc:552] Master@127.15.81.62:35255 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26973 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240502 08:17:19.718647 15684 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.81.62:44475
I20240502 08:17:19.719152 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:19.722493 17806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:19.723523 17807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:19.724257 15684 server_base.cc:1034] running on GCE node
W20240502 08:17:19.724903 17809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:19.725204 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:17:19.725262 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:17:19.725298 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637839725293 us; error 0 us; skew 500 ppm
I20240502 08:17:19.725435 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:19.726598 15684 webserver.cc:460] Webserver started at http://127.15.81.62:38003/ using document root <none> and password file <none>
I20240502 08:17:19.726817 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:19.726881 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:19.727011 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:19.727669 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714637795017386-15684-0/minicluster-data/master-0-root/instance:
uuid: "0cc3d2f08b9f474ebb1b15d8f3694102"
format_stamp: "Formatted at 2024-05-02 08:17:19 on dist-test-slave-rhmj"
I20240502 08:17:19.730057 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:19.731496 17814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:19.731848 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 08:17:19.731946 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714637795017386-15684-0/minicluster-data/master-0-root
uuid: "0cc3d2f08b9f474ebb1b15d8f3694102"
format_stamp: "Formatted at 2024-05-02 08:17:19 on dist-test-slave-rhmj"
I20240502 08:17:19.732053 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714637795017386-15684-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714637795017386-15684-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714637795017386-15684-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:19.743948 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:19.744539 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:19.756547 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.62:44475
I20240502 08:17:19.756569 17866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.62:44475 every 8 connection(s)
I20240502 08:17:19.757809 17867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:19.761719 17867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102: Bootstrap starting.
I20240502 08:17:19.762774 17867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:19.764207 17867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102: No bootstrap required, opened a new log
I20240502 08:17:19.764758 17867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc3d2f08b9f474ebb1b15d8f3694102" member_type: VOTER }
I20240502 08:17:19.764879 17867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:19.764920 17867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc3d2f08b9f474ebb1b15d8f3694102, State: Initialized, Role: FOLLOWER
I20240502 08:17:19.765084 17867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc3d2f08b9f474ebb1b15d8f3694102" member_type: VOTER }
I20240502 08:17:19.765228 17867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:17:19.765264 17867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:17:19.765313 17867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:19.766983 17867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc3d2f08b9f474ebb1b15d8f3694102" member_type: VOTER }
I20240502 08:17:19.767143 17867 leader_election.cc:302] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cc3d2f08b9f474ebb1b15d8f3694102; no voters:
I20240502 08:17:19.767467 17867 leader_election.cc:288] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:17:19.767585 17870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:19.768095 17870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [term 1 LEADER]: Becoming Leader. State: Replica: 0cc3d2f08b9f474ebb1b15d8f3694102, State: Running, Role: LEADER
I20240502 08:17:19.768261 17867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:17:19.768291 17870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc3d2f08b9f474ebb1b15d8f3694102" member_type: VOTER }
I20240502 08:17:19.768976 17872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cc3d2f08b9f474ebb1b15d8f3694102. Latest consensus state: current_term: 1 leader_uuid: "0cc3d2f08b9f474ebb1b15d8f3694102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc3d2f08b9f474ebb1b15d8f3694102" member_type: VOTER } }
I20240502 08:17:19.769168 17871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cc3d2f08b9f474ebb1b15d8f3694102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc3d2f08b9f474ebb1b15d8f3694102" member_type: VOTER } }
I20240502 08:17:19.769248 17872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:19.769343 17871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:19.770287 17875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:17:19.771726 17875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:17:19.773285 15684 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:17:19.774612 17875 catalog_manager.cc:1347] Generated new cluster ID: 57e5a03ff10b4f98a6fc06b4191e0ecf
I20240502 08:17:19.774688 17875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:17:19.790460 17875 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:17:19.791221 17875 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:17:19.800205 17875 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102: Generated new TSK 0
I20240502 08:17:19.800416 17875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:17:19.805706 15684 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 6.6481e-05s
I20240502 08:17:31.893661 15684 master.cc:537] Master@127.15.81.62:44475 shutting down...
I20240502 08:17:31.905869 15684 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:31.906174 15684 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:31.906284 15684 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0cc3d2f08b9f474ebb1b15d8f3694102: stopping tablet replica
I20240502 08:17:31.925336 15684 master.cc:552] Master@127.15.81.62:44475 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12214 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240502 08:17:31.933507 15684 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.81.62:46169
I20240502 08:17:31.934065 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:31.937793 17891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:31.938589 17890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:31.939337 15684 server_base.cc:1034] running on GCE node
W20240502 08:17:31.940253 17893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:31.940583 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:17:31.940654 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:17:31.940711 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637851940691 us; error 0 us; skew 500 ppm
I20240502 08:17:31.940889 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:31.942270 15684 webserver.cc:460] Webserver started at http://127.15.81.62:39875/ using document root <none> and password file <none>
I20240502 08:17:31.942505 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:31.942586 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:31.942735 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:31.943392 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/master-0-root/instance:
uuid: "ded2c7aaf50c45ddb6e8b79769956cc4"
format_stamp: "Formatted at 2024-05-02 08:17:31 on dist-test-slave-rhmj"
I20240502 08:17:31.946017 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:31.947687 17898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:31.948092 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240502 08:17:31.948195 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/master-0-root
uuid: "ded2c7aaf50c45ddb6e8b79769956cc4"
format_stamp: "Formatted at 2024-05-02 08:17:31 on dist-test-slave-rhmj"
I20240502 08:17:31.948309 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:31.972883 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:31.973385 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:31.984441 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.62:46169
I20240502 08:17:31.984460 17949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.62:46169 every 8 connection(s)
I20240502 08:17:31.985683 17950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:31.989648 17950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4: Bootstrap starting.
I20240502 08:17:31.990851 17950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:31.992331 17950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4: No bootstrap required, opened a new log
I20240502 08:17:31.992841 17950 raft_consensus.cc:348] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded2c7aaf50c45ddb6e8b79769956cc4" member_type: VOTER }
I20240502 08:17:31.992966 17950 raft_consensus.cc:374] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:31.993005 17950 raft_consensus.cc:729] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded2c7aaf50c45ddb6e8b79769956cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:31.993147 17950 consensus_queue.cc:260] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded2c7aaf50c45ddb6e8b79769956cc4" member_type: VOTER }
I20240502 08:17:31.993237 17950 raft_consensus.cc:388] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:17:31.993274 17950 raft_consensus.cc:482] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:17:31.993321 17950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:31.994889 17950 raft_consensus.cc:504] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded2c7aaf50c45ddb6e8b79769956cc4" member_type: VOTER }
I20240502 08:17:31.995054 17950 leader_election.cc:302] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ded2c7aaf50c45ddb6e8b79769956cc4; no voters:
I20240502 08:17:31.995380 17950 leader_election.cc:288] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:17:31.995509 17954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:31.995985 17954 raft_consensus.cc:686] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [term 1 LEADER]: Becoming Leader. State: Replica: ded2c7aaf50c45ddb6e8b79769956cc4, State: Running, Role: LEADER
I20240502 08:17:31.996156 17950 sys_catalog.cc:564] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:17:31.996181 17954 consensus_queue.cc:237] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded2c7aaf50c45ddb6e8b79769956cc4" member_type: VOTER }
I20240502 08:17:31.996953 17956 sys_catalog.cc:455] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ded2c7aaf50c45ddb6e8b79769956cc4. Latest consensus state: current_term: 1 leader_uuid: "ded2c7aaf50c45ddb6e8b79769956cc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded2c7aaf50c45ddb6e8b79769956cc4" member_type: VOTER } }
I20240502 08:17:31.997148 17956 sys_catalog.cc:458] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:31.997359 17955 sys_catalog.cc:455] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ded2c7aaf50c45ddb6e8b79769956cc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded2c7aaf50c45ddb6e8b79769956cc4" member_type: VOTER } }
I20240502 08:17:31.997627 17955 sys_catalog.cc:458] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:32.000305 17959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:17:32.001555 17959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:17:32.001919 15684 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:17:32.004463 17959 catalog_manager.cc:1347] Generated new cluster ID: 90dcf1f349994dd58af10d2ebd994255
I20240502 08:17:32.004555 17959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:17:32.020275 17959 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:17:32.021032 17959 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:17:32.029191 17959 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4: Generated new TSK 0
I20240502 08:17:32.029431 17959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:17:32.034708 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:32.038365 17973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:32.039122 17974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:32.039911 17976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:32.040884 15684 server_base.cc:1034] running on GCE node
I20240502 08:17:32.041106 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:17:32.041160 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:17:32.041204 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637852041200 us; error 0 us; skew 500 ppm
I20240502 08:17:32.041340 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:32.042616 15684 webserver.cc:460] Webserver started at http://127.15.81.1:46055/ using document root <none> and password file <none>
I20240502 08:17:32.042834 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:32.042901 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:32.043041 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:32.043695 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/ts-0-root/instance:
uuid: "38d803e75cac4deb82d36de71b97ede6"
format_stamp: "Formatted at 2024-05-02 08:17:32 on dist-test-slave-rhmj"
I20240502 08:17:32.046159 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:32.047593 17981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:32.048034 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:17:32.048131 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/ts-0-root
uuid: "38d803e75cac4deb82d36de71b97ede6"
format_stamp: "Formatted at 2024-05-02 08:17:32 on dist-test-slave-rhmj"
I20240502 08:17:32.048241 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:32.063656 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:32.064244 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:32.064903 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:32.065822 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:32.065881 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:32.065960 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:32.065997 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:32.078527 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.1:35865
I20240502 08:17:32.078541 18043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.1:35865 every 8 connection(s)
I20240502 08:17:32.080169 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:32.083949 18048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:32.084726 18049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:32.089061 18044 heartbeater.cc:347] Connected to a master server at 127.15.81.62:46169
I20240502 08:17:32.089179 18044 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:32.089489 18044 heartbeater.cc:511] Master 127.15.81.62:46169 requested a full tablet report, sending...
I20240502 08:17:32.089732 15684 server_base.cc:1034] running on GCE node
W20240502 08:17:32.090379 18051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:32.090384 17915 ts_manager.cc:194] Registered new tserver with Master: 38d803e75cac4deb82d36de71b97ede6 (127.15.81.1:35865)
I20240502 08:17:32.090736 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:17:32.090785 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:17:32.090811 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637852090804 us; error 0 us; skew 500 ppm
I20240502 08:17:32.090904 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:32.091476 17915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55828
I20240502 08:17:32.092372 15684 webserver.cc:460] Webserver started at http://127.15.81.2:45163/ using document root <none> and password file <none>
I20240502 08:17:32.092582 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:32.092640 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:32.092757 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:32.093358 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/ts-1-root/instance:
uuid: "48e1edf2b34c46b698e2f1c0f2864ab2"
format_stamp: "Formatted at 2024-05-02 08:17:32 on dist-test-slave-rhmj"
I20240502 08:17:32.095724 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:17:32.097206 18056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:32.097512 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:17:32.097597 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/ts-1-root
uuid: "48e1edf2b34c46b698e2f1c0f2864ab2"
format_stamp: "Formatted at 2024-05-02 08:17:32 on dist-test-slave-rhmj"
I20240502 08:17:32.097693 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:32.108039 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:32.108544 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:32.109053 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:32.109794 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:32.109850 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:32.109920 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:32.109963 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:32.124274 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.2:33205
I20240502 08:17:32.124312 18118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.2:33205 every 8 connection(s)
I20240502 08:17:32.125883 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:32.129855 18124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:32.130436 18122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:32.131348 18126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:32.132364 15684 server_base.cc:1034] running on GCE node
I20240502 08:17:32.132651 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:17:32.132711 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:17:32.132745 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637852132740 us; error 0 us; skew 500 ppm
I20240502 08:17:32.132777 18119 heartbeater.cc:347] Connected to a master server at 127.15.81.62:46169
I20240502 08:17:32.132905 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:32.132905 18119 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:32.133333 18119 heartbeater.cc:511] Master 127.15.81.62:46169 requested a full tablet report, sending...
I20240502 08:17:32.134284 17915 ts_manager.cc:194] Registered new tserver with Master: 48e1edf2b34c46b698e2f1c0f2864ab2 (127.15.81.2:33205)
I20240502 08:17:32.134292 15684 webserver.cc:460] Webserver started at http://127.15.81.3:45907/ using document root <none> and password file <none>
I20240502 08:17:32.134521 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:32.134594 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:32.134709 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:32.135085 17915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55832
I20240502 08:17:32.135300 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/ts-2-root/instance:
uuid: "4aa29641f9a54c12aa2d2e834593fbb0"
format_stamp: "Formatted at 2024-05-02 08:17:32 on dist-test-slave-rhmj"
I20240502 08:17:32.137712 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:17:32.139117 18132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:32.139480 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:17:32.139561 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/ts-2-root
uuid: "4aa29641f9a54c12aa2d2e834593fbb0"
format_stamp: "Formatted at 2024-05-02 08:17:32 on dist-test-slave-rhmj"
I20240502 08:17:32.139695 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714637795017386-15684-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:32.161338 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:32.161903 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:32.162461 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:32.163316 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:32.163369 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:32.163439 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:32.163481 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:32.176054 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.3:46861
I20240502 08:17:32.176095 18195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.3:46861 every 8 connection(s)
I20240502 08:17:32.182612 18196 heartbeater.cc:347] Connected to a master server at 127.15.81.62:46169
I20240502 08:17:32.182749 18196 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:32.183086 18196 heartbeater.cc:511] Master 127.15.81.62:46169 requested a full tablet report, sending...
I20240502 08:17:32.183950 17915 ts_manager.cc:194] Registered new tserver with Master: 4aa29641f9a54c12aa2d2e834593fbb0 (127.15.81.3:46861)
I20240502 08:17:32.184618 17915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55842
I20240502 08:17:32.184919 15684 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.0079181s
I20240502 08:17:33.092905 18044 heartbeater.cc:503] Master 127.15.81.62:46169 was elected leader, sending a full tablet report...
I20240502 08:17:33.136305 18119 heartbeater.cc:503] Master 127.15.81.62:46169 was elected leader, sending a full tablet report...
I20240502 08:17:33.185873 18196 heartbeater.cc:503] Master 127.15.81.62:46169 was elected leader, sending a full tablet report...
I20240502 08:17:34.001149 17969 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:36.001667 17969 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:38.002110 17969 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:40.002552 17969 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:42.003006 17969 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:44.003486 17969 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:44.267849 15684 tablet_server.cc:178] TabletServer@127.15.81.1:0 shutting down...
I20240502 08:17:44.280025 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:17:44.294919 15684 tablet_server.cc:194] TabletServer@127.15.81.1:0 shutdown complete.
I20240502 08:17:44.298262 15684 tablet_server.cc:178] TabletServer@127.15.81.2:0 shutting down...
I20240502 08:17:44.307420 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:17:44.320286 15684 tablet_server.cc:194] TabletServer@127.15.81.2:0 shutdown complete.
I20240502 08:17:44.323509 15684 tablet_server.cc:178] TabletServer@127.15.81.3:0 shutting down...
I20240502 08:17:44.332895 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:17:44.345667 15684 tablet_server.cc:194] TabletServer@127.15.81.3:0 shutdown complete.
I20240502 08:17:44.348652 15684 master.cc:537] Master@127.15.81.62:46169 shutting down...
I20240502 08:17:44.356705 15684 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:44.356969 15684 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:44.357100 15684 tablet_replica.cc:330] T 00000000000000000000000000000000 P ded2c7aaf50c45ddb6e8b79769956cc4: stopping tablet replica
I20240502 08:17:44.370435 15684 master.cc:552] Master@127.15.81.62:46169 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12446 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240502 08:17:44.380342 15684 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.81.62:45785
I20240502 08:17:44.380839 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:44.384045 18200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:44.385171 18201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:44.385946 18203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:44.386615 15684 server_base.cc:1034] running on GCE node
I20240502 08:17:44.386904 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:17:44.386978 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:17:44.387043 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637864387020 us; error 0 us; skew 500 ppm
I20240502 08:17:44.387216 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:44.392887 15684 webserver.cc:460] Webserver started at http://127.15.81.62:39113/ using document root <none> and password file <none>
I20240502 08:17:44.393133 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:44.393215 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:44.393360 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:44.394034 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/master-0-root/instance:
uuid: "403e4372138e488498952e5135758173"
format_stamp: "Formatted at 2024-05-02 08:17:44 on dist-test-slave-rhmj"
I20240502 08:17:44.396595 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:17:44.398365 18208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:44.398766 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:17:44.411733 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/master-0-root
uuid: "403e4372138e488498952e5135758173"
format_stamp: "Formatted at 2024-05-02 08:17:44 on dist-test-slave-rhmj"
I20240502 08:17:44.411921 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:44.420626 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:44.421098 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:44.430809 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.62:45785
I20240502 08:17:44.430820 18260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.62:45785 every 8 connection(s)
I20240502 08:17:44.432102 18261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.435992 18261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173: Bootstrap starting.
I20240502 08:17:44.437009 18261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.438328 18261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173: No bootstrap required, opened a new log
I20240502 08:17:44.438825 18261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403e4372138e488498952e5135758173" member_type: VOTER }
I20240502 08:17:44.438951 18261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.438990 18261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403e4372138e488498952e5135758173, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.439131 18261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403e4372138e488498952e5135758173" member_type: VOTER }
I20240502 08:17:44.439220 18261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:17:44.439258 18261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:17:44.439306 18261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:44.440970 18261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403e4372138e488498952e5135758173" member_type: VOTER }
I20240502 08:17:44.441134 18261 leader_election.cc:302] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 403e4372138e488498952e5135758173; no voters:
I20240502 08:17:44.441444 18261 leader_election.cc:288] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:17:44.441560 18264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:44.442013 18264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [term 1 LEADER]: Becoming Leader. State: Replica: 403e4372138e488498952e5135758173, State: Running, Role: LEADER
I20240502 08:17:44.442221 18264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403e4372138e488498952e5135758173" member_type: VOTER }
I20240502 08:17:44.442310 18261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:17:44.442965 18266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 403e4372138e488498952e5135758173. Latest consensus state: current_term: 1 leader_uuid: "403e4372138e488498952e5135758173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403e4372138e488498952e5135758173" member_type: VOTER } }
I20240502 08:17:44.443022 18265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "403e4372138e488498952e5135758173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403e4372138e488498952e5135758173" member_type: VOTER } }
I20240502 08:17:44.443161 18266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:44.443255 18265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:44.447656 15684 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
W20240502 08:17:44.448479 18280 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 08:17:44.448568 18280 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 08:17:44.448665 18269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:17:44.449895 18269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:17:44.452723 18269 catalog_manager.cc:1347] Generated new cluster ID: c2dfee76a8ae4b2095398e17ca3eaa15
I20240502 08:17:44.452807 18269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:17:44.465108 18269 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:17:44.465900 18269 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:17:44.472488 18269 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173: Generated new TSK 0
I20240502 08:17:44.472754 18269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:17:44.480412 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:44.484025 18282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:44.484762 18283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:44.485432 18285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:44.486440 15684 server_base.cc:1034] running on GCE node
I20240502 08:17:44.486653 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:17:44.486713 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:17:44.486740 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637864486737 us; error 0 us; skew 500 ppm
I20240502 08:17:44.486841 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:44.488162 15684 webserver.cc:460] Webserver started at http://127.15.81.1:33833/ using document root <none> and password file <none>
I20240502 08:17:44.488359 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:44.488417 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:44.488505 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:44.489151 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-0-root/instance:
uuid: "b4ce69f5e87a431398e8ebc2d53bc18b"
format_stamp: "Formatted at 2024-05-02 08:17:44 on dist-test-slave-rhmj"
I20240502 08:17:44.491684 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:17:44.493206 18291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:44.493521 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:17:44.493623 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-0-root
uuid: "b4ce69f5e87a431398e8ebc2d53bc18b"
format_stamp: "Formatted at 2024-05-02 08:17:44 on dist-test-slave-rhmj"
I20240502 08:17:44.493739 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:44.519822 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:44.520365 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:44.520951 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:44.521894 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:44.521953 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:44.522024 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:44.522060 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:44.533689 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.1:36681
I20240502 08:17:44.533720 18353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.1:36681 every 8 connection(s)
I20240502 08:17:44.535308 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:44.538906 18358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:44.539783 18359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:44.540442 15684 server_base.cc:1034] running on GCE node
W20240502 08:17:44.542256 18361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:44.542685 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:17:44.542750 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:17:44.542785 18354 heartbeater.cc:347] Connected to a master server at 127.15.81.62:45785
I20240502 08:17:44.542797 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637864542793 us; error 0 us; skew 500 ppm
I20240502 08:17:44.542917 18354 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:44.543025 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:44.543292 18354 heartbeater.cc:511] Master 127.15.81.62:45785 requested a full tablet report, sending...
I20240502 08:17:44.544310 18225 ts_manager.cc:194] Registered new tserver with Master: b4ce69f5e87a431398e8ebc2d53bc18b (127.15.81.1:36681)
I20240502 08:17:44.544616 15684 webserver.cc:460] Webserver started at http://127.15.81.2:45045/ using document root <none> and password file <none>
I20240502 08:17:44.544864 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:44.544940 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:44.545099 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:44.545666 18225 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41498
I20240502 08:17:44.545828 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-1-root/instance:
uuid: "4adbb70074054551a1f5fcd70e513919"
format_stamp: "Formatted at 2024-05-02 08:17:44 on dist-test-slave-rhmj"
I20240502 08:17:44.548720 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240502 08:17:44.550170 18366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:44.550455 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:17:44.550546 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-1-root
uuid: "4adbb70074054551a1f5fcd70e513919"
format_stamp: "Formatted at 2024-05-02 08:17:44 on dist-test-slave-rhmj"
I20240502 08:17:44.550640 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:44.561457 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:44.561955 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:44.562417 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:44.563155 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:44.563210 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:44.563277 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:44.563310 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:44.574792 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.2:35861
I20240502 08:17:44.574800 18430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.2:35861 every 8 connection(s)
I20240502 08:17:44.576385 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:44.580173 18435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:44.581064 18436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:44.581848 15684 server_base.cc:1034] running on GCE node
W20240502 08:17:44.582568 18438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:44.582854 18431 heartbeater.cc:347] Connected to a master server at 127.15.81.62:45785
I20240502 08:17:44.582978 18431 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:44.583007 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:17:44.583086 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:17:44.583127 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637864583122 us; error 0 us; skew 500 ppm
I20240502 08:17:44.583259 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:44.583355 18431 heartbeater.cc:511] Master 127.15.81.62:45785 requested a full tablet report, sending...
I20240502 08:17:44.584208 18225 ts_manager.cc:194] Registered new tserver with Master: 4adbb70074054551a1f5fcd70e513919 (127.15.81.2:35861)
I20240502 08:17:44.584901 18225 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41510
I20240502 08:17:44.585902 15684 webserver.cc:460] Webserver started at http://127.15.81.3:37477/ using document root <none> and password file <none>
I20240502 08:17:44.586125 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:44.586198 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:44.586318 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:44.586923 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-2-root/instance:
uuid: "65ecb729663c48e68ee74d9b655c6525"
format_stamp: "Formatted at 2024-05-02 08:17:44 on dist-test-slave-rhmj"
I20240502 08:17:44.589282 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 08:17:44.590786 18443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:44.591192 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 08:17:44.591287 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-2-root
uuid: "65ecb729663c48e68ee74d9b655c6525"
format_stamp: "Formatted at 2024-05-02 08:17:44 on dist-test-slave-rhmj"
I20240502 08:17:44.591396 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:44.613245 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:44.613848 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:44.614441 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:44.615303 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:44.615361 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:44.615433 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:44.615469 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:44.627797 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.3:38351
I20240502 08:17:44.627820 18510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.3:38351 every 8 connection(s)
I20240502 08:17:44.629452 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:44.633352 18514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:44.634194 18515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:44.635030 18517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:44.635648 18511 heartbeater.cc:347] Connected to a master server at 127.15.81.62:45785
I20240502 08:17:44.635752 15684 server_base.cc:1034] running on GCE node
I20240502 08:17:44.635838 18511 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:44.636090 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:17:44.636168 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:17:44.636245 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637864636214 us; error 0 us; skew 500 ppm
I20240502 08:17:44.636251 18511 heartbeater.cc:511] Master 127.15.81.62:45785 requested a full tablet report, sending...
I20240502 08:17:44.636423 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:44.637180 18225 ts_manager.cc:194] Registered new tserver with Master: 65ecb729663c48e68ee74d9b655c6525 (127.15.81.3:38351)
I20240502 08:17:44.638073 15684 webserver.cc:460] Webserver started at http://127.15.81.4:38631/ using document root <none> and password file <none>
I20240502 08:17:44.638283 18225 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41522
I20240502 08:17:44.638325 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:44.638440 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:44.638597 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:44.639278 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-3-root/instance:
uuid: "c2723820521b4452a4a15de01308e03d"
format_stamp: "Formatted at 2024-05-02 08:17:44 on dist-test-slave-rhmj"
I20240502 08:17:44.641713 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 08:17:44.643389 18523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:44.643793 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240502 08:17:44.643908 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-3-root
uuid: "c2723820521b4452a4a15de01308e03d"
format_stamp: "Formatted at 2024-05-02 08:17:44 on dist-test-slave-rhmj"
I20240502 08:17:44.644032 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714637795017386-15684-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:44.665817 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:44.666424 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:44.667014 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:44.667933 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:44.668015 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:44.668118 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:44.668183 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:44.680646 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.4:35461
I20240502 08:17:44.680671 18588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.4:35461 every 8 connection(s)
I20240502 08:17:44.687031 18589 heartbeater.cc:347] Connected to a master server at 127.15.81.62:45785
I20240502 08:17:44.687173 18589 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:44.687541 18589 heartbeater.cc:511] Master 127.15.81.62:45785 requested a full tablet report, sending...
I20240502 08:17:44.688335 18225 ts_manager.cc:194] Registered new tserver with Master: c2723820521b4452a4a15de01308e03d (127.15.81.4:35461)
I20240502 08:17:44.688408 15684 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.00686442s
I20240502 08:17:44.689391 18225 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41530
I20240502 08:17:45.547210 18354 heartbeater.cc:503] Master 127.15.81.62:45785 was elected leader, sending a full tablet report...
I20240502 08:17:45.586172 18431 heartbeater.cc:503] Master 127.15.81.62:45785 was elected leader, sending a full tablet report...
I20240502 08:17:45.639567 18511 heartbeater.cc:503] Master 127.15.81.62:45785 was elected leader, sending a full tablet report...
I20240502 08:17:45.690582 18589 heartbeater.cc:503] Master 127.15.81.62:45785 was elected leader, sending a full tablet report...
I20240502 08:17:45.723409 15684 test_util.cc:268] Using random seed: 1021814282
I20240502 08:17:45.734685 18225 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41544:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:45.762483 18552 tablet_service.cc:1467] Processing CreateTablet for tablet 8a59fca965574182a208c37007dd5e1c (DEFAULT_TABLE table=test-workload [id=7ada863e063a43f09ffb5dd68d28c485]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:45.763027 18552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a59fca965574182a208c37007dd5e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:45.764329 18319 tablet_service.cc:1467] Processing CreateTablet for tablet 8a59fca965574182a208c37007dd5e1c (DEFAULT_TABLE table=test-workload [id=7ada863e063a43f09ffb5dd68d28c485]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:45.764798 18319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a59fca965574182a208c37007dd5e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:45.767951 18473 tablet_service.cc:1467] Processing CreateTablet for tablet 8a59fca965574182a208c37007dd5e1c (DEFAULT_TABLE table=test-workload [id=7ada863e063a43f09ffb5dd68d28c485]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:45.768374 18473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a59fca965574182a208c37007dd5e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:45.769342 18610 tablet_bootstrap.cc:492] T 8a59fca965574182a208c37007dd5e1c P c2723820521b4452a4a15de01308e03d: Bootstrap starting.
I20240502 08:17:45.771780 18610 tablet_bootstrap.cc:654] T 8a59fca965574182a208c37007dd5e1c P c2723820521b4452a4a15de01308e03d: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:45.774318 18610 tablet_bootstrap.cc:492] T 8a59fca965574182a208c37007dd5e1c P c2723820521b4452a4a15de01308e03d: No bootstrap required, opened a new log
I20240502 08:17:45.774513 18610 ts_tablet_manager.cc:1367] T 8a59fca965574182a208c37007dd5e1c P c2723820521b4452a4a15de01308e03d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 08:17:45.775528 18610 raft_consensus.cc:348] T 8a59fca965574182a208c37007dd5e1c P c2723820521b4452a4a15de01308e03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723820521b4452a4a15de01308e03d" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35461 } } peers { permanent_uuid: "b4ce69f5e87a431398e8ebc2d53bc18b" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36681 } } peers { permanent_uuid: "65ecb729663c48e68ee74d9b655c6525" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 38351 } }
I20240502 08:17:45.775856 18610 raft_consensus.cc:374] T 8a59fca965574182a208c37007dd5e1c P c2723820521b4452a4a15de01308e03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:45.775970 18613 tablet_bootstrap.cc:492] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525: Bootstrap starting.
I20240502 08:17:45.775991 18610 raft_consensus.cc:729] T 8a59fca965574182a208c37007dd5e1c P c2723820521b4452a4a15de01308e03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2723820521b4452a4a15de01308e03d, State: Initialized, Role: FOLLOWER
I20240502 08:17:45.776340 18610 consensus_queue.cc:260] T 8a59fca965574182a208c37007dd5e1c P c2723820521b4452a4a15de01308e03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723820521b4452a4a15de01308e03d" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35461 } } peers { permanent_uuid: "b4ce69f5e87a431398e8ebc2d53bc18b" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36681 } } peers { permanent_uuid: "65ecb729663c48e68ee74d9b655c6525" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 38351 } }
I20240502 08:17:45.777400 18610 ts_tablet_manager.cc:1398] T 8a59fca965574182a208c37007dd5e1c P c2723820521b4452a4a15de01308e03d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:45.777771 18613 tablet_bootstrap.cc:654] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:45.779330 18611 tablet_bootstrap.cc:492] T 8a59fca965574182a208c37007dd5e1c P b4ce69f5e87a431398e8ebc2d53bc18b: Bootstrap starting.
I20240502 08:17:45.780514 18613 tablet_bootstrap.cc:492] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525: No bootstrap required, opened a new log
I20240502 08:17:45.780732 18613 ts_tablet_manager.cc:1367] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 08:17:45.781661 18611 tablet_bootstrap.cc:654] T 8a59fca965574182a208c37007dd5e1c P b4ce69f5e87a431398e8ebc2d53bc18b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:45.781674 18613 raft_consensus.cc:348] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723820521b4452a4a15de01308e03d" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35461 } } peers { permanent_uuid: "b4ce69f5e87a431398e8ebc2d53bc18b" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36681 } } peers { permanent_uuid: "65ecb729663c48e68ee74d9b655c6525" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 38351 } }
I20240502 08:17:45.782064 18613 raft_consensus.cc:374] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:45.782171 18613 raft_consensus.cc:729] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ecb729663c48e68ee74d9b655c6525, State: Initialized, Role: FOLLOWER
I20240502 08:17:45.782444 18613 consensus_queue.cc:260] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723820521b4452a4a15de01308e03d" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35461 } } peers { permanent_uuid: "b4ce69f5e87a431398e8ebc2d53bc18b" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36681 } } peers { permanent_uuid: "65ecb729663c48e68ee74d9b655c6525" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 38351 } }
I20240502 08:17:45.783270 18613 ts_tablet_manager.cc:1398] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:45.783753 18611 tablet_bootstrap.cc:492] T 8a59fca965574182a208c37007dd5e1c P b4ce69f5e87a431398e8ebc2d53bc18b: No bootstrap required, opened a new log
I20240502 08:17:45.783906 18611 ts_tablet_manager.cc:1367] T 8a59fca965574182a208c37007dd5e1c P b4ce69f5e87a431398e8ebc2d53bc18b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:17:45.784662 18611 raft_consensus.cc:348] T 8a59fca965574182a208c37007dd5e1c P b4ce69f5e87a431398e8ebc2d53bc18b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723820521b4452a4a15de01308e03d" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35461 } } peers { permanent_uuid: "b4ce69f5e87a431398e8ebc2d53bc18b" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36681 } } peers { permanent_uuid: "65ecb729663c48e68ee74d9b655c6525" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 38351 } }
I20240502 08:17:45.784833 18611 raft_consensus.cc:374] T 8a59fca965574182a208c37007dd5e1c P b4ce69f5e87a431398e8ebc2d53bc18b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:45.784922 18611 raft_consensus.cc:729] T 8a59fca965574182a208c37007dd5e1c P b4ce69f5e87a431398e8ebc2d53bc18b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ce69f5e87a431398e8ebc2d53bc18b, State: Initialized, Role: FOLLOWER
I20240502 08:17:45.785230 18611 consensus_queue.cc:260] T 8a59fca965574182a208c37007dd5e1c P b4ce69f5e87a431398e8ebc2d53bc18b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723820521b4452a4a15de01308e03d" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35461 } } peers { permanent_uuid: "b4ce69f5e87a431398e8ebc2d53bc18b" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36681 } } peers { permanent_uuid: "65ecb729663c48e68ee74d9b655c6525" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 38351 } }
I20240502 08:17:45.786154 18611 ts_tablet_manager.cc:1398] T 8a59fca965574182a208c37007dd5e1c P b4ce69f5e87a431398e8ebc2d53bc18b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:45.798959 18617 raft_consensus.cc:482] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:45.799151 18617 raft_consensus.cc:504] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723820521b4452a4a15de01308e03d" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35461 } } peers { permanent_uuid: "b4ce69f5e87a431398e8ebc2d53bc18b" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36681 } } peers { permanent_uuid: "65ecb729663c48e68ee74d9b655c6525" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 38351 } }
I20240502 08:17:45.799988 18617 leader_election.cc:288] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2723820521b4452a4a15de01308e03d (127.15.81.4:35461), b4ce69f5e87a431398e8ebc2d53bc18b (127.15.81.1:36681)
I20240502 08:17:45.806759 18563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a59fca965574182a208c37007dd5e1c" candidate_uuid: "65ecb729663c48e68ee74d9b655c6525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2723820521b4452a4a15de01308e03d" is_pre_election: true
I20240502 08:17:45.806780 18329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a59fca965574182a208c37007dd5e1c" candidate_uuid: "65ecb729663c48e68ee74d9b655c6525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4ce69f5e87a431398e8ebc2d53bc18b" is_pre_election: true
I20240502 08:17:45.807018 18563 raft_consensus.cc:2454] T 8a59fca965574182a208c37007dd5e1c P c2723820521b4452a4a15de01308e03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65ecb729663c48e68ee74d9b655c6525 in term 0.
I20240502 08:17:45.807071 18329 raft_consensus.cc:2454] T 8a59fca965574182a208c37007dd5e1c P b4ce69f5e87a431398e8ebc2d53bc18b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65ecb729663c48e68ee74d9b655c6525 in term 0.
I20240502 08:17:45.807416 18445 leader_election.cc:302] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65ecb729663c48e68ee74d9b655c6525, c2723820521b4452a4a15de01308e03d; no voters:
I20240502 08:17:45.807695 18617 raft_consensus.cc:2781] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:45.807786 18617 raft_consensus.cc:482] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:45.807875 18617 raft_consensus.cc:3037] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.809732 18617 raft_consensus.cc:504] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723820521b4452a4a15de01308e03d" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35461 } } peers { permanent_uuid: "b4ce69f5e87a431398e8ebc2d53bc18b" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36681 } } peers { permanent_uuid: "65ecb729663c48e68ee74d9b655c6525" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 38351 } }
I20240502 08:17:45.810299 18617 leader_election.cc:288] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [CANDIDATE]: Term 1 election: Requested vote from peers c2723820521b4452a4a15de01308e03d (127.15.81.4:35461), b4ce69f5e87a431398e8ebc2d53bc18b (127.15.81.1:36681)
I20240502 08:17:45.810746 18563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a59fca965574182a208c37007dd5e1c" candidate_uuid: "65ecb729663c48e68ee74d9b655c6525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2723820521b4452a4a15de01308e03d"
I20240502 08:17:45.810828 18329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a59fca965574182a208c37007dd5e1c" candidate_uuid: "65ecb729663c48e68ee74d9b655c6525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4ce69f5e87a431398e8ebc2d53bc18b"
I20240502 08:17:45.810925 18563 raft_consensus.cc:3037] T 8a59fca965574182a208c37007dd5e1c P c2723820521b4452a4a15de01308e03d [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.811024 18329 raft_consensus.cc:3037] T 8a59fca965574182a208c37007dd5e1c P b4ce69f5e87a431398e8ebc2d53bc18b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.813325 18563 raft_consensus.cc:2454] T 8a59fca965574182a208c37007dd5e1c P c2723820521b4452a4a15de01308e03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65ecb729663c48e68ee74d9b655c6525 in term 1.
I20240502 08:17:45.813390 18329 raft_consensus.cc:2454] T 8a59fca965574182a208c37007dd5e1c P b4ce69f5e87a431398e8ebc2d53bc18b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65ecb729663c48e68ee74d9b655c6525 in term 1.
I20240502 08:17:45.813702 18445 leader_election.cc:302] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65ecb729663c48e68ee74d9b655c6525, c2723820521b4452a4a15de01308e03d; no voters:
I20240502 08:17:45.814018 18617 raft_consensus.cc:2781] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:45.814394 18617 raft_consensus.cc:686] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [term 1 LEADER]: Becoming Leader. State: Replica: 65ecb729663c48e68ee74d9b655c6525, State: Running, Role: LEADER
I20240502 08:17:45.814646 18617 consensus_queue.cc:237] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723820521b4452a4a15de01308e03d" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35461 } } peers { permanent_uuid: "b4ce69f5e87a431398e8ebc2d53bc18b" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36681 } } peers { permanent_uuid: "65ecb729663c48e68ee74d9b655c6525" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 38351 } }
I20240502 08:17:45.817059 18225 catalog_manager.cc:5484] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65ecb729663c48e68ee74d9b655c6525 (127.15.81.3). New cstate: current_term: 1 leader_uuid: "65ecb729663c48e68ee74d9b655c6525" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723820521b4452a4a15de01308e03d" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4ce69f5e87a431398e8ebc2d53bc18b" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65ecb729663c48e68ee74d9b655c6525" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 38351 } health_report { overall_health: HEALTHY } } }
W20240502 08:17:45.879974 18512 tablet.cc:2369] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:46.280401 18617 consensus_queue.cc:1035] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4ce69f5e87a431398e8ebc2d53bc18b" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:46.285583 18621 consensus_queue.cc:1035] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2723820521b4452a4a15de01308e03d" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 35461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:46.447677 18279 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:46.448210 18279 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:46.448356 18279 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:48.448854 18279 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:48.449339 18279 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:48.449441 18279 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:50.449880 18279 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:50.450333 18279 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:50.450440 18279 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:52.450908 18279 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:52.451437 18279 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:52.451596 18279 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:54.452118 18279 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:54.452659 18279 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:54.452775 18279 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:56.453204 18279 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:56.453701 18279 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:56.453809 18279 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:56.880084 15684 tablet_server.cc:178] TabletServer@127.15.81.1:0 shutting down...
I20240502 08:17:56.893748 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:17:56.894140 15684 tablet_replica.cc:330] T 8a59fca965574182a208c37007dd5e1c P b4ce69f5e87a431398e8ebc2d53bc18b: stopping tablet replica
I20240502 08:17:56.894330 15684 raft_consensus.cc:2229] T 8a59fca965574182a208c37007dd5e1c P b4ce69f5e87a431398e8ebc2d53bc18b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:56.894517 15684 raft_consensus.cc:2258] T 8a59fca965574182a208c37007dd5e1c P b4ce69f5e87a431398e8ebc2d53bc18b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.908887 15684 tablet_server.cc:194] TabletServer@127.15.81.1:0 shutdown complete.
I20240502 08:17:56.912065 15684 tablet_server.cc:178] TabletServer@127.15.81.2:0 shutting down...
I20240502 08:17:56.921828 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:17:56.948150 15684 tablet_server.cc:194] TabletServer@127.15.81.2:0 shutdown complete.
I20240502 08:17:56.950954 15684 tablet_server.cc:178] TabletServer@127.15.81.3:0 shutting down...
I20240502 08:17:56.960336 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:17:56.960678 15684 tablet_replica.cc:330] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525: stopping tablet replica
I20240502 08:17:56.960875 15684 raft_consensus.cc:2229] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:56.961194 15684 raft_consensus.cc:2258] T 8a59fca965574182a208c37007dd5e1c P 65ecb729663c48e68ee74d9b655c6525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.975107 15684 tablet_server.cc:194] TabletServer@127.15.81.3:0 shutdown complete.
I20240502 08:17:56.978370 15684 tablet_server.cc:178] TabletServer@127.15.81.4:0 shutting down...
I20240502 08:17:56.986454 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:17:56.986749 15684 tablet_replica.cc:330] T 8a59fca965574182a208c37007dd5e1c P c2723820521b4452a4a15de01308e03d: stopping tablet replica
I20240502 08:17:56.986935 15684 raft_consensus.cc:2229] T 8a59fca965574182a208c37007dd5e1c P c2723820521b4452a4a15de01308e03d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:56.987095 15684 raft_consensus.cc:2258] T 8a59fca965574182a208c37007dd5e1c P c2723820521b4452a4a15de01308e03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:57.000949 15684 tablet_server.cc:194] TabletServer@127.15.81.4:0 shutdown complete.
I20240502 08:17:57.004330 15684 master.cc:537] Master@127.15.81.62:45785 shutting down...
I20240502 08:17:57.011935 15684 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:57.012199 15684 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:57.012298 15684 tablet_replica.cc:330] T 00000000000000000000000000000000 P 403e4372138e488498952e5135758173: stopping tablet replica
I20240502 08:17:57.025709 15684 master.cc:552] Master@127.15.81.62:45785 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12657 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240502 08:17:57.037737 15684 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.81.62:39781
I20240502 08:17:57.038254 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:57.041934 18638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:57.042928 18639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:57.043581 18641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:57.044381 15684 server_base.cc:1034] running on GCE node
I20240502 08:17:57.044622 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:17:57.044677 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:17:57.044698 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637877044695 us; error 0 us; skew 500 ppm
I20240502 08:17:57.044831 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:57.052264 15684 webserver.cc:460] Webserver started at http://127.15.81.62:41193/ using document root <none> and password file <none>
I20240502 08:17:57.052513 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:57.052614 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:57.052727 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:57.053416 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/master-0-root/instance:
uuid: "cea2f457915c45d0b429fcd1196ccbc9"
format_stamp: "Formatted at 2024-05-02 08:17:57 on dist-test-slave-rhmj"
I20240502 08:17:57.056370 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:17:57.058298 18646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:57.058737 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 08:17:57.058835 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/master-0-root
uuid: "cea2f457915c45d0b429fcd1196ccbc9"
format_stamp: "Formatted at 2024-05-02 08:17:57 on dist-test-slave-rhmj"
I20240502 08:17:57.058969 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:57.078588 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:57.079216 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:57.090925 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.62:39781
I20240502 08:17:57.090947 18698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.62:39781 every 8 connection(s)
I20240502 08:17:57.092362 18699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.096484 18699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9: Bootstrap starting.
I20240502 08:17:57.097674 18699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.099227 18699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9: No bootstrap required, opened a new log
I20240502 08:17:57.099870 18699 raft_consensus.cc:348] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea2f457915c45d0b429fcd1196ccbc9" member_type: VOTER }
I20240502 08:17:57.099995 18699 raft_consensus.cc:374] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.100044 18699 raft_consensus.cc:729] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea2f457915c45d0b429fcd1196ccbc9, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.100260 18699 consensus_queue.cc:260] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea2f457915c45d0b429fcd1196ccbc9" member_type: VOTER }
I20240502 08:17:57.100379 18699 raft_consensus.cc:388] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:17:57.100440 18699 raft_consensus.cc:482] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:17:57.100515 18699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.102221 18699 raft_consensus.cc:504] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea2f457915c45d0b429fcd1196ccbc9" member_type: VOTER }
I20240502 08:17:57.102394 18699 leader_election.cc:302] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cea2f457915c45d0b429fcd1196ccbc9; no voters:
I20240502 08:17:57.102773 18699 leader_election.cc:288] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:17:57.102910 18702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:57.103348 18702 raft_consensus.cc:686] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [term 1 LEADER]: Becoming Leader. State: Replica: cea2f457915c45d0b429fcd1196ccbc9, State: Running, Role: LEADER
I20240502 08:17:57.103650 18699 sys_catalog.cc:564] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:17:57.103585 18702 consensus_queue.cc:237] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea2f457915c45d0b429fcd1196ccbc9" member_type: VOTER }
I20240502 08:17:57.104555 18703 sys_catalog.cc:455] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cea2f457915c45d0b429fcd1196ccbc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea2f457915c45d0b429fcd1196ccbc9" member_type: VOTER } }
I20240502 08:17:57.104614 18704 sys_catalog.cc:455] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cea2f457915c45d0b429fcd1196ccbc9. Latest consensus state: current_term: 1 leader_uuid: "cea2f457915c45d0b429fcd1196ccbc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea2f457915c45d0b429fcd1196ccbc9" member_type: VOTER } }
I20240502 08:17:57.104851 18703 sys_catalog.cc:458] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:57.104887 18704 sys_catalog.cc:458] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:57.106392 18707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:17:57.108078 18707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:17:57.108839 15684 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:17:57.111094 18707 catalog_manager.cc:1347] Generated new cluster ID: 1a9346801ece495b8bac298f86414355
I20240502 08:17:57.111166 18707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:17:57.123246 18707 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:17:57.124063 18707 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:17:57.129516 18707 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9: Generated new TSK 0
I20240502 08:17:57.129747 18707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:17:57.141672 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:57.145241 18720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:57.145953 18721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:57.146618 18723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:57.147696 15684 server_base.cc:1034] running on GCE node
I20240502 08:17:57.147933 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:17:57.147989 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:17:57.148039 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637877148035 us; error 0 us; skew 500 ppm
I20240502 08:17:57.148193 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:57.149500 15684 webserver.cc:460] Webserver started at http://127.15.81.1:41287/ using document root <none> and password file <none>
I20240502 08:17:57.149757 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:57.149844 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:57.149981 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:57.150671 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/ts-0-root/instance:
uuid: "4c6c3140edfb4464ba84c66d6b803985"
format_stamp: "Formatted at 2024-05-02 08:17:57 on dist-test-slave-rhmj"
I20240502 08:17:57.153484 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:17:57.155365 18728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:57.155943 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 08:17:57.156044 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/ts-0-root
uuid: "4c6c3140edfb4464ba84c66d6b803985"
format_stamp: "Formatted at 2024-05-02 08:17:57 on dist-test-slave-rhmj"
I20240502 08:17:57.156149 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:57.165724 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:57.166221 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:57.166700 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:57.167562 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:57.167740 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:57.167872 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:57.167912 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:57.180727 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.1:45367
I20240502 08:17:57.180748 18790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.1:45367 every 8 connection(s)
I20240502 08:17:57.182475 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:57.186116 18795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:57.186887 18796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:57.187497 18798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:57.188222 15684 server_base.cc:1034] running on GCE node
I20240502 08:17:57.188535 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:17:57.188604 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:17:57.188632 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637877188628 us; error 0 us; skew 500 ppm
I20240502 08:17:57.188777 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:57.190146 15684 webserver.cc:460] Webserver started at http://127.15.81.2:36063/ using document root <none> and password file <none>
I20240502 08:17:57.190388 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:57.190476 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:57.190599 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:57.190734 18791 heartbeater.cc:347] Connected to a master server at 127.15.81.62:39781
I20240502 08:17:57.190876 18791 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:57.191184 18791 heartbeater.cc:511] Master 127.15.81.62:39781 requested a full tablet report, sending...
I20240502 08:17:57.191249 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/ts-1-root/instance:
uuid: "55320785bb4f4792985a1032bd32126f"
format_stamp: "Formatted at 2024-05-02 08:17:57 on dist-test-slave-rhmj"
I20240502 08:17:57.192224 18664 ts_manager.cc:194] Registered new tserver with Master: 4c6c3140edfb4464ba84c66d6b803985 (127.15.81.1:45367)
I20240502 08:17:57.193773 18664 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40080
I20240502 08:17:57.194379 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:17:57.195986 18804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:57.196352 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 08:17:57.196434 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/ts-1-root
uuid: "55320785bb4f4792985a1032bd32126f"
format_stamp: "Formatted at 2024-05-02 08:17:57 on dist-test-slave-rhmj"
I20240502 08:17:57.196532 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:57.202494 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:57.203056 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:57.203649 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:57.204496 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:57.204562 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:57.204641 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:57.204684 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:57.216629 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.2:36233
I20240502 08:17:57.216653 18872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.2:36233 every 8 connection(s)
I20240502 08:17:57.218367 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:57.222401 18876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:57.223111 18877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:57.224009 18879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:57.224757 15684 server_base.cc:1034] running on GCE node
I20240502 08:17:57.225015 18873 heartbeater.cc:347] Connected to a master server at 127.15.81.62:39781
I20240502 08:17:57.225158 18873 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:57.225214 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:17:57.225284 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:17:57.225324 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637877225320 us; error 0 us; skew 500 ppm
I20240502 08:17:57.225461 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:57.225520 18873 heartbeater.cc:511] Master 127.15.81.62:39781 requested a full tablet report, sending...
I20240502 08:17:57.226444 18664 ts_manager.cc:194] Registered new tserver with Master: 55320785bb4f4792985a1032bd32126f (127.15.81.2:36233)
I20240502 08:17:57.226867 15684 webserver.cc:460] Webserver started at http://127.15.81.3:34149/ using document root <none> and password file <none>
I20240502 08:17:57.227068 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:57.227137 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:57.227252 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:57.227339 18664 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40092
I20240502 08:17:57.228106 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/ts-2-root/instance:
uuid: "2499c1c26b92460daa9c8433551c77d9"
format_stamp: "Formatted at 2024-05-02 08:17:57 on dist-test-slave-rhmj"
I20240502 08:17:57.230631 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:17:57.232321 18885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:57.232683 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:17:57.232792 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/ts-2-root
uuid: "2499c1c26b92460daa9c8433551c77d9"
format_stamp: "Formatted at 2024-05-02 08:17:57 on dist-test-slave-rhmj"
I20240502 08:17:57.232896 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714637795017386-15684-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:57.243557 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:57.244117 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:57.244690 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:57.245541 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:57.245627 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:57.245733 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:57.245798 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:57.257766 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.3:46621
I20240502 08:17:57.257787 18949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.3:46621 every 8 connection(s)
I20240502 08:17:57.264339 18950 heartbeater.cc:347] Connected to a master server at 127.15.81.62:39781
I20240502 08:17:57.264524 18950 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:57.264951 18950 heartbeater.cc:511] Master 127.15.81.62:39781 requested a full tablet report, sending...
I20240502 08:17:57.265923 18664 ts_manager.cc:194] Registered new tserver with Master: 2499c1c26b92460daa9c8433551c77d9 (127.15.81.3:46621)
I20240502 08:17:57.266673 18664 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40094
I20240502 08:17:57.266785 15684 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008101878s
I20240502 08:17:58.195155 18791 heartbeater.cc:503] Master 127.15.81.62:39781 was elected leader, sending a full tablet report...
I20240502 08:17:58.228541 18873 heartbeater.cc:503] Master 127.15.81.62:39781 was elected leader, sending a full tablet report...
I20240502 08:17:58.267887 18950 heartbeater.cc:503] Master 127.15.81.62:39781 was elected leader, sending a full tablet report...
I20240502 08:17:58.301946 15684 test_util.cc:268] Using random seed: 1034392803
I20240502 08:17:58.313437 18664 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40108:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:58.314044 18664 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:58.336834 18756 tablet_service.cc:1467] Processing CreateTablet for tablet 33e0fa1c5e5f463784aa3daf2be29ebc (DEFAULT_TABLE table=test-workload [id=0c0ce1baf7ca4972b7c8b310ac559c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.337313 18756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e0fa1c5e5f463784aa3daf2be29ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.337769 18835 tablet_service.cc:1467] Processing CreateTablet for tablet 33e0fa1c5e5f463784aa3daf2be29ebc (DEFAULT_TABLE table=test-workload [id=0c0ce1baf7ca4972b7c8b310ac559c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.338158 18835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e0fa1c5e5f463784aa3daf2be29ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.341599 18915 tablet_service.cc:1467] Processing CreateTablet for tablet 33e0fa1c5e5f463784aa3daf2be29ebc (DEFAULT_TABLE table=test-workload [id=0c0ce1baf7ca4972b7c8b310ac559c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.341984 18915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e0fa1c5e5f463784aa3daf2be29ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.343299 18971 tablet_bootstrap.cc:492] T 33e0fa1c5e5f463784aa3daf2be29ebc P 4c6c3140edfb4464ba84c66d6b803985: Bootstrap starting.
I20240502 08:17:58.344390 18972 tablet_bootstrap.cc:492] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f: Bootstrap starting.
I20240502 08:17:58.344857 18971 tablet_bootstrap.cc:654] T 33e0fa1c5e5f463784aa3daf2be29ebc P 4c6c3140edfb4464ba84c66d6b803985: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:58.345947 18972 tablet_bootstrap.cc:654] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:58.346763 18971 tablet_bootstrap.cc:492] T 33e0fa1c5e5f463784aa3daf2be29ebc P 4c6c3140edfb4464ba84c66d6b803985: No bootstrap required, opened a new log
I20240502 08:17:58.346917 18971 ts_tablet_manager.cc:1367] T 33e0fa1c5e5f463784aa3daf2be29ebc P 4c6c3140edfb4464ba84c66d6b803985: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:58.347681 18972 tablet_bootstrap.cc:492] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f: No bootstrap required, opened a new log
I20240502 08:17:58.347672 18971 raft_consensus.cc:348] T 33e0fa1c5e5f463784aa3daf2be29ebc P 4c6c3140edfb4464ba84c66d6b803985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6c3140edfb4464ba84c66d6b803985" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 45367 } } peers { permanent_uuid: "2499c1c26b92460daa9c8433551c77d9" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 46621 } } peers { permanent_uuid: "55320785bb4f4792985a1032bd32126f" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 36233 } }
I20240502 08:17:58.347864 18972 ts_tablet_manager.cc:1367] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:58.347903 18971 raft_consensus.cc:374] T 33e0fa1c5e5f463784aa3daf2be29ebc P 4c6c3140edfb4464ba84c66d6b803985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:58.347992 18971 raft_consensus.cc:729] T 33e0fa1c5e5f463784aa3daf2be29ebc P 4c6c3140edfb4464ba84c66d6b803985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6c3140edfb4464ba84c66d6b803985, State: Initialized, Role: FOLLOWER
I20240502 08:17:58.348402 18975 tablet_bootstrap.cc:492] T 33e0fa1c5e5f463784aa3daf2be29ebc P 2499c1c26b92460daa9c8433551c77d9: Bootstrap starting.
I20240502 08:17:58.348304 18971 consensus_queue.cc:260] T 33e0fa1c5e5f463784aa3daf2be29ebc P 4c6c3140edfb4464ba84c66d6b803985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6c3140edfb4464ba84c66d6b803985" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 45367 } } peers { permanent_uuid: "2499c1c26b92460daa9c8433551c77d9" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 46621 } } peers { permanent_uuid: "55320785bb4f4792985a1032bd32126f" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 36233 } }
I20240502 08:17:58.348523 18972 raft_consensus.cc:348] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6c3140edfb4464ba84c66d6b803985" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 45367 } } peers { permanent_uuid: "2499c1c26b92460daa9c8433551c77d9" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 46621 } } peers { permanent_uuid: "55320785bb4f4792985a1032bd32126f" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 36233 } }
I20240502 08:17:58.348726 18972 raft_consensus.cc:374] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:58.348814 18972 raft_consensus.cc:729] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55320785bb4f4792985a1032bd32126f, State: Initialized, Role: FOLLOWER
I20240502 08:17:58.349072 18972 consensus_queue.cc:260] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6c3140edfb4464ba84c66d6b803985" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 45367 } } peers { permanent_uuid: "2499c1c26b92460daa9c8433551c77d9" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 46621 } } peers { permanent_uuid: "55320785bb4f4792985a1032bd32126f" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 36233 } }
I20240502 08:17:58.349709 18975 tablet_bootstrap.cc:654] T 33e0fa1c5e5f463784aa3daf2be29ebc P 2499c1c26b92460daa9c8433551c77d9: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:58.349980 18972 ts_tablet_manager.cc:1398] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:58.351222 18975 tablet_bootstrap.cc:492] T 33e0fa1c5e5f463784aa3daf2be29ebc P 2499c1c26b92460daa9c8433551c77d9: No bootstrap required, opened a new log
I20240502 08:17:58.351343 18975 ts_tablet_manager.cc:1367] T 33e0fa1c5e5f463784aa3daf2be29ebc P 2499c1c26b92460daa9c8433551c77d9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:58.352149 18975 raft_consensus.cc:348] T 33e0fa1c5e5f463784aa3daf2be29ebc P 2499c1c26b92460daa9c8433551c77d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6c3140edfb4464ba84c66d6b803985" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 45367 } } peers { permanent_uuid: "2499c1c26b92460daa9c8433551c77d9" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 46621 } } peers { permanent_uuid: "55320785bb4f4792985a1032bd32126f" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 36233 } }
I20240502 08:17:58.352384 18975 raft_consensus.cc:374] T 33e0fa1c5e5f463784aa3daf2be29ebc P 2499c1c26b92460daa9c8433551c77d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:58.352490 18975 raft_consensus.cc:729] T 33e0fa1c5e5f463784aa3daf2be29ebc P 2499c1c26b92460daa9c8433551c77d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2499c1c26b92460daa9c8433551c77d9, State: Initialized, Role: FOLLOWER
I20240502 08:17:58.352790 18975 consensus_queue.cc:260] T 33e0fa1c5e5f463784aa3daf2be29ebc P 2499c1c26b92460daa9c8433551c77d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6c3140edfb4464ba84c66d6b803985" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 45367 } } peers { permanent_uuid: "2499c1c26b92460daa9c8433551c77d9" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 46621 } } peers { permanent_uuid: "55320785bb4f4792985a1032bd32126f" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 36233 } }
I20240502 08:17:58.353530 18975 ts_tablet_manager.cc:1398] T 33e0fa1c5e5f463784aa3daf2be29ebc P 2499c1c26b92460daa9c8433551c77d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:58.355659 18971 ts_tablet_manager.cc:1398] T 33e0fa1c5e5f463784aa3daf2be29ebc P 4c6c3140edfb4464ba84c66d6b803985: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20240502 08:17:58.580365 18977 raft_consensus.cc:482] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:58.580574 18977 raft_consensus.cc:504] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6c3140edfb4464ba84c66d6b803985" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 45367 } } peers { permanent_uuid: "2499c1c26b92460daa9c8433551c77d9" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 46621 } } peers { permanent_uuid: "55320785bb4f4792985a1032bd32126f" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 36233 } }
I20240502 08:17:58.581423 18977 leader_election.cc:288] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c6c3140edfb4464ba84c66d6b803985 (127.15.81.1:45367), 2499c1c26b92460daa9c8433551c77d9 (127.15.81.3:46621)
I20240502 08:17:58.587406 18766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e0fa1c5e5f463784aa3daf2be29ebc" candidate_uuid: "55320785bb4f4792985a1032bd32126f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6c3140edfb4464ba84c66d6b803985" is_pre_election: true
I20240502 08:17:58.587509 18925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e0fa1c5e5f463784aa3daf2be29ebc" candidate_uuid: "55320785bb4f4792985a1032bd32126f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2499c1c26b92460daa9c8433551c77d9" is_pre_election: true
I20240502 08:17:58.587674 18766 raft_consensus.cc:2454] T 33e0fa1c5e5f463784aa3daf2be29ebc P 4c6c3140edfb4464ba84c66d6b803985 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55320785bb4f4792985a1032bd32126f in term 0.
I20240502 08:17:58.587755 18925 raft_consensus.cc:2454] T 33e0fa1c5e5f463784aa3daf2be29ebc P 2499c1c26b92460daa9c8433551c77d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55320785bb4f4792985a1032bd32126f in term 0.
I20240502 08:17:58.588106 18808 leader_election.cc:302] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6c3140edfb4464ba84c66d6b803985, 55320785bb4f4792985a1032bd32126f; no voters:
I20240502 08:17:58.588352 18977 raft_consensus.cc:2781] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:58.588421 18977 raft_consensus.cc:482] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:58.588480 18977 raft_consensus.cc:3037] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:58.590493 18977 raft_consensus.cc:504] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6c3140edfb4464ba84c66d6b803985" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 45367 } } peers { permanent_uuid: "2499c1c26b92460daa9c8433551c77d9" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 46621 } } peers { permanent_uuid: "55320785bb4f4792985a1032bd32126f" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 36233 } }
I20240502 08:17:58.591020 18977 leader_election.cc:288] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [CANDIDATE]: Term 1 election: Requested vote from peers 4c6c3140edfb4464ba84c66d6b803985 (127.15.81.1:45367), 2499c1c26b92460daa9c8433551c77d9 (127.15.81.3:46621)
I20240502 08:17:58.591514 18766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e0fa1c5e5f463784aa3daf2be29ebc" candidate_uuid: "55320785bb4f4792985a1032bd32126f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6c3140edfb4464ba84c66d6b803985"
I20240502 08:17:58.591514 18925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e0fa1c5e5f463784aa3daf2be29ebc" candidate_uuid: "55320785bb4f4792985a1032bd32126f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2499c1c26b92460daa9c8433551c77d9"
I20240502 08:17:58.591775 18766 raft_consensus.cc:3037] T 33e0fa1c5e5f463784aa3daf2be29ebc P 4c6c3140edfb4464ba84c66d6b803985 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:58.591809 18925 raft_consensus.cc:3037] T 33e0fa1c5e5f463784aa3daf2be29ebc P 2499c1c26b92460daa9c8433551c77d9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:58.593726 18766 raft_consensus.cc:2454] T 33e0fa1c5e5f463784aa3daf2be29ebc P 4c6c3140edfb4464ba84c66d6b803985 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55320785bb4f4792985a1032bd32126f in term 1.
I20240502 08:17:58.593740 18925 raft_consensus.cc:2454] T 33e0fa1c5e5f463784aa3daf2be29ebc P 2499c1c26b92460daa9c8433551c77d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55320785bb4f4792985a1032bd32126f in term 1.
I20240502 08:17:58.594126 18808 leader_election.cc:302] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6c3140edfb4464ba84c66d6b803985, 55320785bb4f4792985a1032bd32126f; no voters:
I20240502 08:17:58.594383 18977 raft_consensus.cc:2781] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:58.594795 18977 raft_consensus.cc:686] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [term 1 LEADER]: Becoming Leader. State: Replica: 55320785bb4f4792985a1032bd32126f, State: Running, Role: LEADER
I20240502 08:17:58.595027 18977 consensus_queue.cc:237] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6c3140edfb4464ba84c66d6b803985" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 45367 } } peers { permanent_uuid: "2499c1c26b92460daa9c8433551c77d9" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 46621 } } peers { permanent_uuid: "55320785bb4f4792985a1032bd32126f" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 36233 } }
I20240502 08:17:58.597468 18664 catalog_manager.cc:5484] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f reported cstate change: term changed from 0 to 1, leader changed from <none> to 55320785bb4f4792985a1032bd32126f (127.15.81.2). New cstate: current_term: 1 leader_uuid: "55320785bb4f4792985a1032bd32126f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6c3140edfb4464ba84c66d6b803985" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 45367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2499c1c26b92460daa9c8433551c77d9" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 46621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55320785bb4f4792985a1032bd32126f" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 36233 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:59.108350 18712 placement_policy_util.cc:407] tablet 33e0fa1c5e5f463784aa3daf2be29ebc: detected majority of replicas (2 of 3) at location L0
I20240502 08:17:59.108816 18717 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:17:59.109289 18717 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:17:59.109403 18717 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:17:59.153613 18984 consensus_queue.cc:1035] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2499c1c26b92460daa9c8433551c77d9" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 46621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:59.161492 18987 consensus_queue.cc:1035] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c6c3140edfb4464ba84c66d6b803985" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:00.111322 18712 placement_policy_util.cc:407] tablet 33e0fa1c5e5f463784aa3daf2be29ebc: detected majority of replicas (2 of 3) at location L0
I20240502 08:18:01.109869 18717 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:18:01.110340 18717 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:18:01.110435 18717 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:18:01.113858 18712 placement_policy_util.cc:407] tablet 33e0fa1c5e5f463784aa3daf2be29ebc: detected majority of replicas (2 of 3) at location L0
I20240502 08:18:02.116447 18712 placement_policy_util.cc:407] tablet 33e0fa1c5e5f463784aa3daf2be29ebc: detected majority of replicas (2 of 3) at location L0
I20240502 08:18:03.110828 18717 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:18:03.111282 18717 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:18:03.111379 18717 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:18:03.119028 18712 placement_policy_util.cc:407] tablet 33e0fa1c5e5f463784aa3daf2be29ebc: detected majority of replicas (2 of 3) at location L0
I20240502 08:18:04.121582 18712 placement_policy_util.cc:407] tablet 33e0fa1c5e5f463784aa3daf2be29ebc: detected majority of replicas (2 of 3) at location L0
I20240502 08:18:05.111884 18717 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:18:05.112459 18717 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:18:05.112562 18717 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:18:05.124290 18712 placement_policy_util.cc:407] tablet 33e0fa1c5e5f463784aa3daf2be29ebc: detected majority of replicas (2 of 3) at location L0
I20240502 08:18:06.126757 18712 placement_policy_util.cc:407] tablet 33e0fa1c5e5f463784aa3daf2be29ebc: detected majority of replicas (2 of 3) at location L0
I20240502 08:18:07.112962 18717 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:18:07.113386 18717 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:18:07.113488 18717 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:18:07.129328 18712 placement_policy_util.cc:407] tablet 33e0fa1c5e5f463784aa3daf2be29ebc: detected majority of replicas (2 of 3) at location L0
I20240502 08:18:08.132005 18712 placement_policy_util.cc:407] tablet 33e0fa1c5e5f463784aa3daf2be29ebc: detected majority of replicas (2 of 3) at location L0
I20240502 08:18:09.113929 18717 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:18:09.114369 18717 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:18:09.114457 18717 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:18:09.134516 18712 placement_policy_util.cc:407] tablet 33e0fa1c5e5f463784aa3daf2be29ebc: detected majority of replicas (2 of 3) at location L0
I20240502 08:18:09.684201 15684 tablet_server.cc:178] TabletServer@127.15.81.1:0 shutting down...
I20240502 08:18:09.699533 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:18:09.699908 15684 tablet_replica.cc:330] T 33e0fa1c5e5f463784aa3daf2be29ebc P 4c6c3140edfb4464ba84c66d6b803985: stopping tablet replica
I20240502 08:18:09.700085 15684 raft_consensus.cc:2229] T 33e0fa1c5e5f463784aa3daf2be29ebc P 4c6c3140edfb4464ba84c66d6b803985 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:09.700280 15684 raft_consensus.cc:2258] T 33e0fa1c5e5f463784aa3daf2be29ebc P 4c6c3140edfb4464ba84c66d6b803985 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:09.714210 15684 tablet_server.cc:194] TabletServer@127.15.81.1:0 shutdown complete.
I20240502 08:18:09.717167 15684 tablet_server.cc:178] TabletServer@127.15.81.2:0 shutting down...
I20240502 08:18:09.725795 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:18:09.726054 15684 tablet_replica.cc:330] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f: stopping tablet replica
I20240502 08:18:09.726241 15684 raft_consensus.cc:2229] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:09.726534 15684 raft_consensus.cc:2258] T 33e0fa1c5e5f463784aa3daf2be29ebc P 55320785bb4f4792985a1032bd32126f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:09.740208 15684 tablet_server.cc:194] TabletServer@127.15.81.2:0 shutdown complete.
I20240502 08:18:09.743104 15684 tablet_server.cc:178] TabletServer@127.15.81.3:0 shutting down...
I20240502 08:18:09.751004 15684 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:18:09.751246 15684 tablet_replica.cc:330] T 33e0fa1c5e5f463784aa3daf2be29ebc P 2499c1c26b92460daa9c8433551c77d9: stopping tablet replica
I20240502 08:18:09.751432 15684 raft_consensus.cc:2229] T 33e0fa1c5e5f463784aa3daf2be29ebc P 2499c1c26b92460daa9c8433551c77d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:09.751655 15684 raft_consensus.cc:2258] T 33e0fa1c5e5f463784aa3daf2be29ebc P 2499c1c26b92460daa9c8433551c77d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:09.765146 15684 tablet_server.cc:194] TabletServer@127.15.81.3:0 shutdown complete.
I20240502 08:18:09.768134 15684 master.cc:537] Master@127.15.81.62:39781 shutting down...
I20240502 08:18:09.775327 15684 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:09.775568 15684 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:09.775717 15684 tablet_replica.cc:330] T 00000000000000000000000000000000 P cea2f457915c45d0b429fcd1196ccbc9: stopping tablet replica
I20240502 08:18:09.789676 15684 master.cc:552] Master@127.15.81.62:39781 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12773 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240502 08:18:09.811112 15684 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.81.62:39387
I20240502 08:18:09.811654 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:09.814855 18998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:09.815460 18999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:09.816179 19002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:09.817471 15684 server_base.cc:1034] running on GCE node
I20240502 08:18:09.817729 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:18:09.817783 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:18:09.817821 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637889817816 us; error 0 us; skew 500 ppm
I20240502 08:18:09.817974 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:09.819942 15684 webserver.cc:460] Webserver started at http://127.15.81.62:44917/ using document root <none> and password file <none>
I20240502 08:18:09.820211 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:09.820286 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:09.820433 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:09.821357 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/master-0-root/instance:
uuid: "7336f1ea195a444bbf482f3a7801df03"
format_stamp: "Formatted at 2024-05-02 08:18:09 on dist-test-slave-rhmj"
I20240502 08:18:09.824543 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:18:09.826263 19007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:09.826680 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 08:18:09.826781 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/master-0-root
uuid: "7336f1ea195a444bbf482f3a7801df03"
format_stamp: "Formatted at 2024-05-02 08:18:09 on dist-test-slave-rhmj"
I20240502 08:18:09.826891 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:09.836802 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:18:09.837311 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:09.848261 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.62:39387
I20240502 08:18:09.848274 19059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.62:39387 every 8 connection(s)
I20240502 08:18:09.849526 19060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.853560 19060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03: Bootstrap starting.
I20240502 08:18:09.854620 19060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.857489 19060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03: No bootstrap required, opened a new log
I20240502 08:18:09.858119 19060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7336f1ea195a444bbf482f3a7801df03" member_type: VOTER }
I20240502 08:18:09.858240 19060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.858294 19060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7336f1ea195a444bbf482f3a7801df03, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.858479 19060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7336f1ea195a444bbf482f3a7801df03" member_type: VOTER }
I20240502 08:18:09.858585 19060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:18:09.858639 19060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:18:09.858709 19060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.860424 19060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7336f1ea195a444bbf482f3a7801df03" member_type: VOTER }
I20240502 08:18:09.860594 19060 leader_election.cc:302] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7336f1ea195a444bbf482f3a7801df03; no voters:
I20240502 08:18:09.860991 19060 leader_election.cc:288] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:18:09.861110 19063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:09.861527 19063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [term 1 LEADER]: Becoming Leader. State: Replica: 7336f1ea195a444bbf482f3a7801df03, State: Running, Role: LEADER
I20240502 08:18:09.861737 19063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7336f1ea195a444bbf482f3a7801df03" member_type: VOTER }
I20240502 08:18:09.861794 19060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:18:09.862390 19064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7336f1ea195a444bbf482f3a7801df03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7336f1ea195a444bbf482f3a7801df03" member_type: VOTER } }
I20240502 08:18:09.862434 19065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7336f1ea195a444bbf482f3a7801df03. Latest consensus state: current_term: 1 leader_uuid: "7336f1ea195a444bbf482f3a7801df03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7336f1ea195a444bbf482f3a7801df03" member_type: VOTER } }
I20240502 08:18:09.862586 19065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [sys.catalog]: This master's current role is: LEADER
I20240502 08:18:09.866811 15684 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:18:09.866827 19064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03 [sys.catalog]: This master's current role is: LEADER
W20240502 08:18:09.867581 19079 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 08:18:09.867707 19079 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 08:18:09.867782 19068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:18:09.868801 19068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:18:09.871407 19068 catalog_manager.cc:1347] Generated new cluster ID: 140c7c5bc82a4305a491bac709998324
I20240502 08:18:09.871484 19068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:18:09.885349 19068 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:18:09.886138 19068 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:18:09.899634 19068 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7336f1ea195a444bbf482f3a7801df03: Generated new TSK 0
I20240502 08:18:09.899915 19068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:18:09.932010 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:09.935504 19081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:09.936200 19082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:09.937040 19084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:09.937909 15684 server_base.cc:1034] running on GCE node
I20240502 08:18:09.938195 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:18:09.938253 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:18:09.938282 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637889938276 us; error 0 us; skew 500 ppm
I20240502 08:18:09.938424 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:09.939929 15684 webserver.cc:460] Webserver started at http://127.15.81.1:45457/ using document root <none> and password file <none>
I20240502 08:18:09.940150 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:09.940215 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:09.940347 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:09.941038 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-0-root/instance:
uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
format_stamp: "Formatted at 2024-05-02 08:18:09 on dist-test-slave-rhmj"
I20240502 08:18:09.943711 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 08:18:09.945395 19089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:09.945813 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 08:18:09.945916 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-0-root
uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
format_stamp: "Formatted at 2024-05-02 08:18:09 on dist-test-slave-rhmj"
I20240502 08:18:09.946020 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:09.954501 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:18:09.955003 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:09.955482 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:09.956346 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:09.956401 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:09.956480 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:09.956517 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:09.968645 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.1:36079
I20240502 08:18:09.968672 19152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.1:36079 every 8 connection(s)
I20240502 08:18:09.970155 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:09.973837 19157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:09.974287 19158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:09.978109 19153 heartbeater.cc:347] Connected to a master server at 127.15.81.62:39387
I20240502 08:18:09.978224 19153 heartbeater.cc:464] Registering TS with master...
W20240502 08:18:09.978518 19160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:09.978547 19153 heartbeater.cc:511] Master 127.15.81.62:39387 requested a full tablet report, sending...
I20240502 08:18:09.979285 15684 server_base.cc:1034] running on GCE node
I20240502 08:18:09.979489 19024 ts_manager.cc:194] Registered new tserver with Master: 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079)
I20240502 08:18:09.979593 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:18:09.979674 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:18:09.979705 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637889979700 us; error 0 us; skew 500 ppm
I20240502 08:18:09.979836 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:09.980960 19024 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33104
I20240502 08:18:09.981189 15684 webserver.cc:460] Webserver started at http://127.15.81.2:44307/ using document root <none> and password file <none>
I20240502 08:18:09.981415 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:09.981498 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:09.981629 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:09.982197 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-1-root/instance:
uuid: "287e8970c9524c998a437fbb3349fb60"
format_stamp: "Formatted at 2024-05-02 08:18:09 on dist-test-slave-rhmj"
I20240502 08:18:09.984609 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:18:09.986176 19165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:09.986593 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 08:18:09.986701 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-1-root
uuid: "287e8970c9524c998a437fbb3349fb60"
format_stamp: "Formatted at 2024-05-02 08:18:09 on dist-test-slave-rhmj"
I20240502 08:18:09.986805 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:10.005370 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:18:10.005980 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:10.006567 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:10.007443 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:10.007500 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:10.007586 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:10.007654 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:10.021209 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.2:38313
I20240502 08:18:10.021235 19228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.2:38313 every 8 connection(s)
I20240502 08:18:10.022804 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:10.026621 19233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:10.027055 19232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:10.027848 19235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:10.028782 15684 server_base.cc:1034] running on GCE node
I20240502 08:18:10.029081 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:18:10.029157 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:18:10.029201 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637890029195 us; error 0 us; skew 500 ppm
I20240502 08:18:10.029211 19229 heartbeater.cc:347] Connected to a master server at 127.15.81.62:39387
I20240502 08:18:10.029321 19229 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:10.029359 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:10.029708 19229 heartbeater.cc:511] Master 127.15.81.62:39387 requested a full tablet report, sending...
I20240502 08:18:10.030782 15684 webserver.cc:460] Webserver started at http://127.15.81.3:46725/ using document root <none> and password file <none>
I20240502 08:18:10.030794 19024 ts_manager.cc:194] Registered new tserver with Master: 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:10.031005 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:10.031065 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:10.031177 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:10.031672 19024 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33114
I20240502 08:18:10.031879 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-2-root/instance:
uuid: "7d576eb536a94138b805dc5ef8b0800c"
format_stamp: "Formatted at 2024-05-02 08:18:10 on dist-test-slave-rhmj"
I20240502 08:18:10.034844 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:18:10.036500 19240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:10.036876 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:18:10.036978 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-2-root
uuid: "7d576eb536a94138b805dc5ef8b0800c"
format_stamp: "Formatted at 2024-05-02 08:18:10 on dist-test-slave-rhmj"
I20240502 08:18:10.037081 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:10.049206 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:18:10.049774 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:10.050299 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:10.051137 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:10.051192 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:10.051272 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:10.051307 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:10.063232 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.3:40169
I20240502 08:18:10.063256 19303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.3:40169 every 8 connection(s)
I20240502 08:18:10.070219 19304 heartbeater.cc:347] Connected to a master server at 127.15.81.62:39387
I20240502 08:18:10.070348 19304 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:10.070745 19304 heartbeater.cc:511] Master 127.15.81.62:39387 requested a full tablet report, sending...
I20240502 08:18:10.071674 19024 ts_manager.cc:194] Registered new tserver with Master: 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:10.072059 15684 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007936936s
I20240502 08:18:10.072489 19024 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33126
I20240502 08:18:10.982303 19153 heartbeater.cc:503] Master 127.15.81.62:39387 was elected leader, sending a full tablet report...
I20240502 08:18:11.033192 19229 heartbeater.cc:503] Master 127.15.81.62:39387 was elected leader, sending a full tablet report...
I20240502 08:18:11.073993 19304 heartbeater.cc:503] Master 127.15.81.62:39387 was elected leader, sending a full tablet report...
I20240502 08:18:11.100229 15684 test_util.cc:268] Using random seed: 1047191101
I20240502 08:18:11.112035 19024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33128:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:18:11.112865 19024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:18:11.147655 19194 tablet_service.cc:1467] Processing CreateTablet for tablet 6a21d56d23364d6f802d89fbc035c02f (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 08:18:11.147898 19269 tablet_service.cc:1467] Processing CreateTablet for tablet 6a21d56d23364d6f802d89fbc035c02f (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 08:18:11.148341 19194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a21d56d23364d6f802d89fbc035c02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.148375 19269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a21d56d23364d6f802d89fbc035c02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.148553 19186 tablet_service.cc:1467] Processing CreateTablet for tablet 4febdd8875b64cc1a156527854a1167a (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 08:18:11.148969 19186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4febdd8875b64cc1a156527854a1167a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.152029 19193 tablet_service.cc:1467] Processing CreateTablet for tablet e844ad43672042da8eabbc9b74e622f7 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 08:18:11.152345 19187 tablet_service.cc:1467] Processing CreateTablet for tablet db356e6346d948938202e72df77e8199 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 08:18:11.153023 19188 tablet_service.cc:1467] Processing CreateTablet for tablet 8fa1524bb0a749f9998928d10f4adea6 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 08:18:11.153765 19189 tablet_service.cc:1467] Processing CreateTablet for tablet b48005466a39437b937e2d09c1826d27 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 08:18:11.154453 19190 tablet_service.cc:1467] Processing CreateTablet for tablet 47ec01cade6e41d1a0ff6c8929ee8155 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 08:18:11.155160 19267 tablet_service.cc:1467] Processing CreateTablet for tablet e844ad43672042da8eabbc9b74e622f7 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 08:18:11.155200 19191 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7253eb5fb84187805cff227d030f54 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 08:18:11.155382 19186 tablet_service.cc:1467] Processing CreateTablet for tablet 5990222a54794361819bebe9e51582e5 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 08:18:11.155545 19267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e844ad43672042da8eabbc9b74e622f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.155956 19261 tablet_service.cc:1467] Processing CreateTablet for tablet db356e6346d948938202e72df77e8199 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 08:18:11.156386 19261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db356e6346d948938202e72df77e8199. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.156425 19192 tablet_service.cc:1467] Processing CreateTablet for tablet 2f640265eb3f4b10a0675e3b08b11680 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 08:18:11.152029 19185 tablet_service.cc:1467] Processing CreateTablet for tablet d3591d08cc604b349553a185225ef943 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 08:18:11.152455 19193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e844ad43672042da8eabbc9b74e622f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.159138 19186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5990222a54794361819bebe9e51582e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.159994 19266 tablet_service.cc:1467] Processing CreateTablet for tablet 2f640265eb3f4b10a0675e3b08b11680 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 08:18:11.160379 19266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f640265eb3f4b10a0675e3b08b11680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.162452 19187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db356e6346d948938202e72df77e8199. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.163180 19259 tablet_service.cc:1467] Processing CreateTablet for tablet d3591d08cc604b349553a185225ef943 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 08:18:11.163599 19259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3591d08cc604b349553a185225ef943. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.163542 19265 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7253eb5fb84187805cff227d030f54 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 08:18:11.164003 19265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7253eb5fb84187805cff227d030f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.165930 19186 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa05fba592744fdbfb354d227626c26 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 08:18:11.166326 19186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa05fba592744fdbfb354d227626c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.166838 19188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fa1524bb0a749f9998928d10f4adea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.167991 19264 tablet_service.cc:1467] Processing CreateTablet for tablet 47ec01cade6e41d1a0ff6c8929ee8155 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 08:18:11.168449 19264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ec01cade6e41d1a0ff6c8929ee8155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.171056 19261 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa05fba592744fdbfb354d227626c26 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 08:18:11.171425 19261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa05fba592744fdbfb354d227626c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.171912 19189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b48005466a39437b937e2d09c1826d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.171988 19263 tablet_service.cc:1467] Processing CreateTablet for tablet b48005466a39437b937e2d09c1826d27 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 08:18:11.172231 19259 tablet_service.cc:1467] Processing CreateTablet for tablet 5990222a54794361819bebe9e51582e5 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 08:18:11.172372 19263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b48005466a39437b937e2d09c1826d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.172595 19259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5990222a54794361819bebe9e51582e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.173044 19262 tablet_service.cc:1467] Processing CreateTablet for tablet 8fa1524bb0a749f9998928d10f4adea6 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 08:18:11.173460 19262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fa1524bb0a749f9998928d10f4adea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.175908 19190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ec01cade6e41d1a0ff6c8929ee8155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.163151 19260 tablet_service.cc:1467] Processing CreateTablet for tablet 4febdd8875b64cc1a156527854a1167a (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 08:18:11.179678 19260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4febdd8875b64cc1a156527854a1167a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.180821 19191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7253eb5fb84187805cff227d030f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.183902 19185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3591d08cc604b349553a185225ef943. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.188226 19192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f640265eb3f4b10a0675e3b08b11680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.193152 19117 tablet_service.cc:1467] Processing CreateTablet for tablet 6a21d56d23364d6f802d89fbc035c02f (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 08:18:11.193627 19117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a21d56d23364d6f802d89fbc035c02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.196543 19116 tablet_service.cc:1467] Processing CreateTablet for tablet e844ad43672042da8eabbc9b74e622f7 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 08:18:11.197036 19116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e844ad43672042da8eabbc9b74e622f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.204726 19114 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7253eb5fb84187805cff227d030f54 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 08:18:11.205492 19114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7253eb5fb84187805cff227d030f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.204730 19115 tablet_service.cc:1467] Processing CreateTablet for tablet 2f640265eb3f4b10a0675e3b08b11680 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 08:18:11.208423 19115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f640265eb3f4b10a0675e3b08b11680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.216594 19113 tablet_service.cc:1467] Processing CreateTablet for tablet 47ec01cade6e41d1a0ff6c8929ee8155 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 08:18:11.219187 19113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ec01cade6e41d1a0ff6c8929ee8155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.217659 19112 tablet_service.cc:1467] Processing CreateTablet for tablet b48005466a39437b937e2d09c1826d27 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 08:18:11.224303 19112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b48005466a39437b937e2d09c1826d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.217782 19110 tablet_service.cc:1467] Processing CreateTablet for tablet db356e6346d948938202e72df77e8199 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 08:18:11.228233 19110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db356e6346d948938202e72df77e8199. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.218472 19111 tablet_service.cc:1467] Processing CreateTablet for tablet 8fa1524bb0a749f9998928d10f4adea6 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 08:18:11.231583 19109 tablet_service.cc:1467] Processing CreateTablet for tablet 5990222a54794361819bebe9e51582e5 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 08:18:11.232132 19111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fa1524bb0a749f9998928d10f4adea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.232489 19109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5990222a54794361819bebe9e51582e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.244187 19116 tablet_service.cc:1467] Processing CreateTablet for tablet d3591d08cc604b349553a185225ef943 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 08:18:11.244927 19116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3591d08cc604b349553a185225ef943. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.246749 19109 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa05fba592744fdbfb354d227626c26 (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 08:18:11.256364 19109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa05fba592744fdbfb354d227626c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.257287 19114 tablet_service.cc:1467] Processing CreateTablet for tablet 4febdd8875b64cc1a156527854a1167a (DEFAULT_TABLE table=test-workload [id=a1cee04baea8496ba25d89318502a3d4]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 08:18:11.262562 19114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4febdd8875b64cc1a156527854a1167a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.270403 19324 tablet_bootstrap.cc:492] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c: Bootstrap starting.
I20240502 08:18:11.276986 19324 tablet_bootstrap.cc:654] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.283208 19325 tablet_bootstrap.cc:492] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60: Bootstrap starting.
I20240502 08:18:11.286522 19325 tablet_bootstrap.cc:654] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.287660 19326 tablet_bootstrap.cc:492] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154: Bootstrap starting.
I20240502 08:18:11.288741 19324 tablet_bootstrap.cc:492] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c: No bootstrap required, opened a new log
I20240502 08:18:11.289031 19324 ts_tablet_manager.cc:1367] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20240502 08:18:11.289762 19326 tablet_bootstrap.cc:654] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.289774 19324 raft_consensus.cc:348] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.290104 19324 raft_consensus.cc:374] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.290238 19325 tablet_bootstrap.cc:492] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60: No bootstrap required, opened a new log
I20240502 08:18:11.290210 19324 raft_consensus.cc:729] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.290398 19325 ts_tablet_manager.cc:1367] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 08:18:11.290565 19324 consensus_queue.cc:260] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.291177 19325 raft_consensus.cc:348] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.291390 19325 raft_consensus.cc:374] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.291479 19325 raft_consensus.cc:729] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.291541 19324 ts_tablet_manager.cc:1398] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:11.291821 19325 consensus_queue.cc:260] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.292254 19326 tablet_bootstrap.cc:492] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154: No bootstrap required, opened a new log
I20240502 08:18:11.292407 19326 ts_tablet_manager.cc:1367] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 08:18:11.292455 19324 tablet_bootstrap.cc:492] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c: Bootstrap starting.
I20240502 08:18:11.293097 19326 raft_consensus.cc:348] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.293318 19326 raft_consensus.cc:374] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.293403 19326 raft_consensus.cc:729] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.293680 19326 consensus_queue.cc:260] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.294700 19324 tablet_bootstrap.cc:654] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.297381 19326 ts_tablet_manager.cc:1398] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240502 08:18:11.297746 19326 tablet_bootstrap.cc:492] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154: Bootstrap starting.
I20240502 08:18:11.298856 19325 ts_tablet_manager.cc:1398] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60: Time spent starting tablet: real 0.008s user 0.000s sys 0.009s
I20240502 08:18:11.299134 19325 tablet_bootstrap.cc:492] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60: Bootstrap starting.
I20240502 08:18:11.299674 19326 tablet_bootstrap.cc:654] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.300964 19325 tablet_bootstrap.cc:654] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.303325 19324 tablet_bootstrap.cc:492] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c: No bootstrap required, opened a new log
I20240502 08:18:11.303534 19324 ts_tablet_manager.cc:1367] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240502 08:18:11.304533 19324 raft_consensus.cc:348] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.304904 19324 raft_consensus.cc:374] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.305038 19324 raft_consensus.cc:729] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.305384 19324 consensus_queue.cc:260] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.306079 19324 ts_tablet_manager.cc:1398] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:11.306422 19324 tablet_bootstrap.cc:492] T d3591d08cc604b349553a185225ef943 P 7d576eb536a94138b805dc5ef8b0800c: Bootstrap starting.
I20240502 08:18:11.308276 19326 tablet_bootstrap.cc:492] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154: No bootstrap required, opened a new log
I20240502 08:18:11.308440 19326 ts_tablet_manager.cc:1367] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 08:18:11.308737 19324 tablet_bootstrap.cc:654] T d3591d08cc604b349553a185225ef943 P 7d576eb536a94138b805dc5ef8b0800c: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.309294 19326 raft_consensus.cc:348] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.309568 19326 raft_consensus.cc:374] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.309660 19326 raft_consensus.cc:729] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.309904 19326 consensus_queue.cc:260] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.310457 19326 ts_tablet_manager.cc:1398] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:11.310767 19326 tablet_bootstrap.cc:492] T e844ad43672042da8eabbc9b74e622f7 P 86029a0bbf0b4f77b9c0e96cf121e154: Bootstrap starting.
I20240502 08:18:11.312444 19326 tablet_bootstrap.cc:654] T e844ad43672042da8eabbc9b74e622f7 P 86029a0bbf0b4f77b9c0e96cf121e154: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.315580 19325 tablet_bootstrap.cc:492] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60: No bootstrap required, opened a new log
I20240502 08:18:11.315783 19325 ts_tablet_manager.cc:1367] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20240502 08:18:11.315886 19324 tablet_bootstrap.cc:492] T d3591d08cc604b349553a185225ef943 P 7d576eb536a94138b805dc5ef8b0800c: No bootstrap required, opened a new log
I20240502 08:18:11.316033 19324 ts_tablet_manager.cc:1367] T d3591d08cc604b349553a185225ef943 P 7d576eb536a94138b805dc5ef8b0800c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240502 08:18:11.316123 19326 tablet_bootstrap.cc:492] T e844ad43672042da8eabbc9b74e622f7 P 86029a0bbf0b4f77b9c0e96cf121e154: No bootstrap required, opened a new log
I20240502 08:18:11.316285 19326 ts_tablet_manager.cc:1367] T e844ad43672042da8eabbc9b74e622f7 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:18:11.316569 19325 raft_consensus.cc:348] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.316741 19324 raft_consensus.cc:348] T d3591d08cc604b349553a185225ef943 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.316865 19325 raft_consensus.cc:374] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.316963 19325 raft_consensus.cc:729] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.316927 19326 raft_consensus.cc:348] T e844ad43672042da8eabbc9b74e622f7 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.316929 19324 raft_consensus.cc:374] T d3591d08cc604b349553a185225ef943 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.317176 19326 raft_consensus.cc:374] T e844ad43672042da8eabbc9b74e622f7 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.317307 19324 raft_consensus.cc:729] T d3591d08cc604b349553a185225ef943 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.317299 19325 consensus_queue.cc:260] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.317312 19326 raft_consensus.cc:729] T e844ad43672042da8eabbc9b74e622f7 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.317581 19324 consensus_queue.cc:260] T d3591d08cc604b349553a185225ef943 P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.317721 19326 consensus_queue.cc:260] T e844ad43672042da8eabbc9b74e622f7 P 86029a0bbf0b4f77b9c0e96cf121e154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.317978 19325 ts_tablet_manager.cc:1398] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:11.318149 19324 ts_tablet_manager.cc:1398] T d3591d08cc604b349553a185225ef943 P 7d576eb536a94138b805dc5ef8b0800c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:11.318253 19325 tablet_bootstrap.cc:492] T 5990222a54794361819bebe9e51582e5 P 287e8970c9524c998a437fbb3349fb60: Bootstrap starting.
I20240502 08:18:11.318390 19326 ts_tablet_manager.cc:1398] T e844ad43672042da8eabbc9b74e622f7 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:11.318418 19324 tablet_bootstrap.cc:492] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c: Bootstrap starting.
I20240502 08:18:11.318702 19326 tablet_bootstrap.cc:492] T 7d7253eb5fb84187805cff227d030f54 P 86029a0bbf0b4f77b9c0e96cf121e154: Bootstrap starting.
I20240502 08:18:11.320109 19325 tablet_bootstrap.cc:654] T 5990222a54794361819bebe9e51582e5 P 287e8970c9524c998a437fbb3349fb60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.320111 19324 tablet_bootstrap.cc:654] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.320489 19326 tablet_bootstrap.cc:654] T 7d7253eb5fb84187805cff227d030f54 P 86029a0bbf0b4f77b9c0e96cf121e154: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.328423 19324 tablet_bootstrap.cc:492] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c: No bootstrap required, opened a new log
I20240502 08:18:11.328505 19325 tablet_bootstrap.cc:492] T 5990222a54794361819bebe9e51582e5 P 287e8970c9524c998a437fbb3349fb60: No bootstrap required, opened a new log
I20240502 08:18:11.328593 19324 ts_tablet_manager.cc:1367] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240502 08:18:11.328677 19325 ts_tablet_manager.cc:1367] T 5990222a54794361819bebe9e51582e5 P 287e8970c9524c998a437fbb3349fb60: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.001s
I20240502 08:18:11.328889 19326 tablet_bootstrap.cc:492] T 7d7253eb5fb84187805cff227d030f54 P 86029a0bbf0b4f77b9c0e96cf121e154: No bootstrap required, opened a new log
I20240502 08:18:11.329034 19326 ts_tablet_manager.cc:1367] T 7d7253eb5fb84187805cff227d030f54 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20240502 08:18:11.329331 19324 raft_consensus.cc:348] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.329514 19324 raft_consensus.cc:374] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.329602 19324 raft_consensus.cc:729] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.329483 19325 raft_consensus.cc:348] T 5990222a54794361819bebe9e51582e5 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.329723 19325 raft_consensus.cc:374] T 5990222a54794361819bebe9e51582e5 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.329658 19326 raft_consensus.cc:348] T 7d7253eb5fb84187805cff227d030f54 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.329813 19325 raft_consensus.cc:729] T 5990222a54794361819bebe9e51582e5 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.329869 19326 raft_consensus.cc:374] T 7d7253eb5fb84187805cff227d030f54 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.329965 19326 raft_consensus.cc:729] T 7d7253eb5fb84187805cff227d030f54 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.329875 19324 consensus_queue.cc:260] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.330106 19325 consensus_queue.cc:260] T 5990222a54794361819bebe9e51582e5 P 287e8970c9524c998a437fbb3349fb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.330232 19326 consensus_queue.cc:260] T 7d7253eb5fb84187805cff227d030f54 P 86029a0bbf0b4f77b9c0e96cf121e154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.330659 19325 ts_tablet_manager.cc:1398] T 5990222a54794361819bebe9e51582e5 P 287e8970c9524c998a437fbb3349fb60: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:11.330947 19325 tablet_bootstrap.cc:492] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60: Bootstrap starting.
I20240502 08:18:11.331045 19324 ts_tablet_manager.cc:1398] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:11.331334 19324 tablet_bootstrap.cc:492] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c: Bootstrap starting.
I20240502 08:18:11.331544 19326 ts_tablet_manager.cc:1398] T 7d7253eb5fb84187805cff227d030f54 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:11.331910 19326 tablet_bootstrap.cc:492] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154: Bootstrap starting.
I20240502 08:18:11.332930 19325 tablet_bootstrap.cc:654] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.333209 19324 tablet_bootstrap.cc:654] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.333588 19326 tablet_bootstrap.cc:654] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.341468 19324 tablet_bootstrap.cc:492] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c: No bootstrap required, opened a new log
I20240502 08:18:11.341660 19324 ts_tablet_manager.cc:1367] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240502 08:18:11.342155 19326 tablet_bootstrap.cc:492] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154: No bootstrap required, opened a new log
I20240502 08:18:11.342365 19326 ts_tablet_manager.cc:1367] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240502 08:18:11.342527 19324 raft_consensus.cc:348] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.342803 19324 raft_consensus.cc:374] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.342908 19324 raft_consensus.cc:729] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.343223 19324 consensus_queue.cc:260] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.343317 19326 raft_consensus.cc:348] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.343592 19326 raft_consensus.cc:374] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.343719 19326 raft_consensus.cc:729] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.343871 19324 ts_tablet_manager.cc:1398] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:11.344192 19324 tablet_bootstrap.cc:492] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c: Bootstrap starting.
I20240502 08:18:11.344040 19326 consensus_queue.cc:260] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.344776 19325 tablet_bootstrap.cc:492] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60: No bootstrap required, opened a new log
I20240502 08:18:11.345042 19325 ts_tablet_manager.cc:1367] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20240502 08:18:11.345829 19325 raft_consensus.cc:348] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.346055 19324 tablet_bootstrap.cc:654] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.346134 19325 raft_consensus.cc:374] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.346349 19325 raft_consensus.cc:729] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.346729 19325 consensus_queue.cc:260] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.346884 19326 ts_tablet_manager.cc:1398] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240502 08:18:11.347390 19325 ts_tablet_manager.cc:1398] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:11.347765 19325 tablet_bootstrap.cc:492] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60: Bootstrap starting.
I20240502 08:18:11.349682 19326 tablet_bootstrap.cc:492] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154: Bootstrap starting.
I20240502 08:18:11.350138 19325 tablet_bootstrap.cc:654] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.351722 19326 tablet_bootstrap.cc:654] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.366726 19326 tablet_bootstrap.cc:492] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154: No bootstrap required, opened a new log
I20240502 08:18:11.366911 19326 ts_tablet_manager.cc:1367] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20240502 08:18:11.367077 19324 tablet_bootstrap.cc:492] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c: No bootstrap required, opened a new log
I20240502 08:18:11.367130 19325 tablet_bootstrap.cc:492] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60: No bootstrap required, opened a new log
I20240502 08:18:11.367272 19324 ts_tablet_manager.cc:1367] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.006s
I20240502 08:18:11.367311 19325 ts_tablet_manager.cc:1367] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.011s
I20240502 08:18:11.367669 19326 raft_consensus.cc:348] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.367869 19326 raft_consensus.cc:374] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.367950 19326 raft_consensus.cc:729] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.368083 19324 raft_consensus.cc:348] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.368201 19326 consensus_queue.cc:260] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.368160 19325 raft_consensus.cc:348] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.368321 19324 raft_consensus.cc:374] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.368479 19324 raft_consensus.cc:729] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.368484 19325 raft_consensus.cc:374] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.368728 19326 ts_tablet_manager.cc:1398] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:11.368732 19325 raft_consensus.cc:729] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.368800 19324 consensus_queue.cc:260] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.369071 19326 tablet_bootstrap.cc:492] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154: Bootstrap starting.
I20240502 08:18:11.369272 19325 consensus_queue.cc:260] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.369415 19324 ts_tablet_manager.cc:1398] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:11.369913 19325 ts_tablet_manager.cc:1398] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:11.370276 19325 tablet_bootstrap.cc:492] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60: Bootstrap starting.
I20240502 08:18:11.370374 19324 tablet_bootstrap.cc:492] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c: Bootstrap starting.
I20240502 08:18:11.371003 19326 tablet_bootstrap.cc:654] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.372237 19325 tablet_bootstrap.cc:654] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.372236 19324 tablet_bootstrap.cc:654] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.374657 19325 tablet_bootstrap.cc:492] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60: No bootstrap required, opened a new log
I20240502 08:18:11.374809 19325 ts_tablet_manager.cc:1367] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:18:11.374912 19326 tablet_bootstrap.cc:492] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154: No bootstrap required, opened a new log
I20240502 08:18:11.375068 19326 ts_tablet_manager.cc:1367] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240502 08:18:11.375541 19325 raft_consensus.cc:348] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.375809 19325 raft_consensus.cc:374] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.375922 19325 raft_consensus.cc:729] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.375821 19326 raft_consensus.cc:348] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.376052 19326 raft_consensus.cc:374] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.376149 19326 raft_consensus.cc:729] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.376183 19325 consensus_queue.cc:260] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.376372 19326 consensus_queue.cc:260] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.376765 19325 ts_tablet_manager.cc:1398] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:11.376935 19326 ts_tablet_manager.cc:1398] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:11.377065 19325 tablet_bootstrap.cc:492] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60: Bootstrap starting.
I20240502 08:18:11.377197 19326 tablet_bootstrap.cc:492] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154: Bootstrap starting.
I20240502 08:18:11.378829 19326 tablet_bootstrap.cc:654] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.378870 19325 tablet_bootstrap.cc:654] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.380787 19324 tablet_bootstrap.cc:492] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c: No bootstrap required, opened a new log
I20240502 08:18:11.380991 19326 tablet_bootstrap.cc:492] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154: No bootstrap required, opened a new log
I20240502 08:18:11.381012 19324 ts_tablet_manager.cc:1367] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 08:18:11.381155 19326 ts_tablet_manager.cc:1367] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:11.382014 19326 raft_consensus.cc:348] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.381947 19324 raft_consensus.cc:348] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.382200 19326 raft_consensus.cc:374] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.382268 19324 raft_consensus.cc:374] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.382295 19326 raft_consensus.cc:729] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.382385 19324 raft_consensus.cc:729] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.382536 19326 consensus_queue.cc:260] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.382637 19324 consensus_queue.cc:260] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.382793 19325 tablet_bootstrap.cc:492] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60: No bootstrap required, opened a new log
I20240502 08:18:11.382941 19325 ts_tablet_manager.cc:1367] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:18:11.383179 19324 ts_tablet_manager.cc:1398] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:11.383467 19324 tablet_bootstrap.cc:492] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c: Bootstrap starting.
I20240502 08:18:11.383646 19325 raft_consensus.cc:348] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.383865 19325 raft_consensus.cc:374] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.383952 19325 raft_consensus.cc:729] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.384172 19325 consensus_queue.cc:260] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.384778 19325 ts_tablet_manager.cc:1398] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:11.385080 19325 tablet_bootstrap.cc:492] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60: Bootstrap starting.
I20240502 08:18:11.385248 19324 tablet_bootstrap.cc:654] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.385527 19326 ts_tablet_manager.cc:1398] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:11.385856 19326 tablet_bootstrap.cc:492] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154: Bootstrap starting.
I20240502 08:18:11.387377 19325 tablet_bootstrap.cc:654] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.387900 19326 tablet_bootstrap.cc:654] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.391471 19324 tablet_bootstrap.cc:492] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c: No bootstrap required, opened a new log
I20240502 08:18:11.391654 19324 ts_tablet_manager.cc:1367] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 08:18:11.391765 19325 tablet_bootstrap.cc:492] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60: No bootstrap required, opened a new log
I20240502 08:18:11.391974 19325 ts_tablet_manager.cc:1367] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240502 08:18:11.392318 19324 raft_consensus.cc:348] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.392506 19324 raft_consensus.cc:374] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.392603 19324 raft_consensus.cc:729] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.392843 19324 consensus_queue.cc:260] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.392995 19325 raft_consensus.cc:348] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.393273 19325 raft_consensus.cc:374] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.393371 19325 raft_consensus.cc:729] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.393471 19324 ts_tablet_manager.cc:1398] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:11.393779 19324 tablet_bootstrap.cc:492] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c: Bootstrap starting.
I20240502 08:18:11.394238 19325 consensus_queue.cc:260] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.394631 19326 tablet_bootstrap.cc:492] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154: No bootstrap required, opened a new log
I20240502 08:18:11.394798 19326 ts_tablet_manager.cc:1367] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 08:18:11.395502 19324 tablet_bootstrap.cc:654] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.395568 19325 ts_tablet_manager.cc:1398] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:11.395933 19325 tablet_bootstrap.cc:492] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60: Bootstrap starting.
I20240502 08:18:11.397739 19325 tablet_bootstrap.cc:654] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.399119 19330 raft_consensus.cc:482] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.399300 19330 raft_consensus.cc:504] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.400025 19326 raft_consensus.cc:348] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.400527 19330 leader_election.cc:288] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.400262 19326 raft_consensus.cc:374] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.400734 19326 raft_consensus.cc:729] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.400974 19325 tablet_bootstrap.cc:492] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60: No bootstrap required, opened a new log
I20240502 08:18:11.401134 19325 ts_tablet_manager.cc:1367] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:18:11.401049 19326 consensus_queue.cc:260] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.401908 19325 raft_consensus.cc:348] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.402132 19325 raft_consensus.cc:374] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.402238 19325 raft_consensus.cc:729] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.402523 19325 consensus_queue.cc:260] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.404021 19325 ts_tablet_manager.cc:1398] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:11.404373 19325 tablet_bootstrap.cc:492] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60: Bootstrap starting.
I20240502 08:18:11.405320 19324 tablet_bootstrap.cc:492] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c: No bootstrap required, opened a new log
I20240502 08:18:11.405589 19324 ts_tablet_manager.cc:1367] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.001s
I20240502 08:18:11.406309 19325 tablet_bootstrap.cc:654] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.406656 19324 raft_consensus.cc:348] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.407013 19324 raft_consensus.cc:374] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.407167 19324 raft_consensus.cc:729] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.407637 19326 ts_tablet_manager.cc:1398] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240502 08:18:11.407524 19324 consensus_queue.cc:260] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.408277 19324 ts_tablet_manager.cc:1398] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:11.408624 19324 tablet_bootstrap.cc:492] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c: Bootstrap starting.
I20240502 08:18:11.410990 19324 tablet_bootstrap.cc:654] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.410957 19326 tablet_bootstrap.cc:492] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154: Bootstrap starting.
I20240502 08:18:11.413862 19326 tablet_bootstrap.cc:654] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.414498 19325 tablet_bootstrap.cc:492] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60: No bootstrap required, opened a new log
I20240502 08:18:11.414685 19325 ts_tablet_manager.cc:1367] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 08:18:11.415549 19325 raft_consensus.cc:348] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.415904 19325 raft_consensus.cc:374] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.416023 19325 raft_consensus.cc:729] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.416513 19325 consensus_queue.cc:260] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.417624 19325 ts_tablet_manager.cc:1398] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:18:11.418035 19325 tablet_bootstrap.cc:492] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60: Bootstrap starting.
I20240502 08:18:11.425076 19325 tablet_bootstrap.cc:654] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.425722 19324 tablet_bootstrap.cc:492] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c: No bootstrap required, opened a new log
I20240502 08:18:11.425920 19324 ts_tablet_manager.cc:1367] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.012s
I20240502 08:18:11.426209 19326 tablet_bootstrap.cc:492] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154: No bootstrap required, opened a new log
I20240502 08:18:11.426379 19326 ts_tablet_manager.cc:1367] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.012s
I20240502 08:18:11.426909 19324 raft_consensus.cc:348] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.427157 19324 raft_consensus.cc:374] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.427095 19326 raft_consensus.cc:348] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.427264 19324 raft_consensus.cc:729] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.427317 19326 raft_consensus.cc:374] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.427399 19326 raft_consensus.cc:729] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.427682 19326 consensus_queue.cc:260] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.427628 19324 consensus_queue.cc:260] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.429077 19326 ts_tablet_manager.cc:1398] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:11.429392 19326 tablet_bootstrap.cc:492] T d3591d08cc604b349553a185225ef943 P 86029a0bbf0b4f77b9c0e96cf121e154: Bootstrap starting.
I20240502 08:18:11.430560 19324 ts_tablet_manager.cc:1398] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:11.430876 19324 tablet_bootstrap.cc:492] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c: Bootstrap starting.
I20240502 08:18:11.434396 19326 tablet_bootstrap.cc:654] T d3591d08cc604b349553a185225ef943 P 86029a0bbf0b4f77b9c0e96cf121e154: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.435223 19324 tablet_bootstrap.cc:654] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.435222 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a21d56d23364d6f802d89fbc035c02f" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" is_pre_election: true
I20240502 08:18:11.435892 19127 raft_consensus.cc:2454] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 0.
I20240502 08:18:11.436556 19242 leader_election.cc:302] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:11.436693 19325 tablet_bootstrap.cc:492] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60: No bootstrap required, opened a new log
I20240502 08:18:11.436906 19330 raft_consensus.cc:2781] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.437047 19330 raft_consensus.cc:482] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.437173 19330 raft_consensus.cc:3037] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.440726 19204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a21d56d23364d6f802d89fbc035c02f" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60" is_pre_election: true
I20240502 08:18:11.441159 19204 raft_consensus.cc:2454] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 0.
I20240502 08:18:11.441298 19330 raft_consensus.cc:504] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.442736 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a21d56d23364d6f802d89fbc035c02f" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:11.443017 19127 raft_consensus.cc:3037] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.444574 19204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a21d56d23364d6f802d89fbc035c02f" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:11.444833 19204 raft_consensus.cc:3037] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.449040 19127 raft_consensus.cc:2454] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 1.
I20240502 08:18:11.449445 19204 raft_consensus.cc:2454] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 1.
I20240502 08:18:11.449715 19242 leader_election.cc:302] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:11.436893 19325 ts_tablet_manager.cc:1367] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.007s
I20240502 08:18:11.451561 19325 raft_consensus.cc:348] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.452018 19325 raft_consensus.cc:374] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.452186 19325 raft_consensus.cc:729] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.452570 19325 consensus_queue.cc:260] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.454190 19325 ts_tablet_manager.cc:1398] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:11.454625 19325 tablet_bootstrap.cc:492] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60: Bootstrap starting.
I20240502 08:18:11.449721 19330 leader_election.cc:288] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.458379 19330 raft_consensus.cc:482] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.458529 19330 raft_consensus.cc:504] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.459350 19330 leader_election.cc:288] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.459581 19330 raft_consensus.cc:2781] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.459806 19330 raft_consensus.cc:686] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Becoming Leader. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:11.460177 19330 consensus_queue.cc:237] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.464368 19325 tablet_bootstrap.cc:654] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.468923 19326 tablet_bootstrap.cc:492] T d3591d08cc604b349553a185225ef943 P 86029a0bbf0b4f77b9c0e96cf121e154: No bootstrap required, opened a new log
I20240502 08:18:11.469154 19326 ts_tablet_manager.cc:1367] T d3591d08cc604b349553a185225ef943 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.017s
I20240502 08:18:11.470220 19326 raft_consensus.cc:348] T d3591d08cc604b349553a185225ef943 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.470592 19326 raft_consensus.cc:374] T d3591d08cc604b349553a185225ef943 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.470685 19326 raft_consensus.cc:729] T d3591d08cc604b349553a185225ef943 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.471050 19326 consensus_queue.cc:260] T d3591d08cc604b349553a185225ef943 P 86029a0bbf0b4f77b9c0e96cf121e154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.474829 19324 tablet_bootstrap.cc:492] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c: No bootstrap required, opened a new log
I20240502 08:18:11.475056 19324 ts_tablet_manager.cc:1367] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.021s
I20240502 08:18:11.475457 19326 ts_tablet_manager.cc:1398] T d3591d08cc604b349553a185225ef943 P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:18:11.476141 19324 raft_consensus.cc:348] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.476433 19324 raft_consensus.cc:374] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.476536 19324 raft_consensus.cc:729] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.476821 19324 consensus_queue.cc:260] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.477409 19324 ts_tablet_manager.cc:1398] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:11.478001 19330 raft_consensus.cc:482] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.478260 19324 tablet_bootstrap.cc:492] T 5990222a54794361819bebe9e51582e5 P 7d576eb536a94138b805dc5ef8b0800c: Bootstrap starting.
I20240502 08:18:11.478163 19330 raft_consensus.cc:504] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.478806 19330 leader_election.cc:288] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.478931 19326 tablet_bootstrap.cc:492] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154: Bootstrap starting.
I20240502 08:18:11.480537 19324 tablet_bootstrap.cc:654] T 5990222a54794361819bebe9e51582e5 P 7d576eb536a94138b805dc5ef8b0800c: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.481264 19326 tablet_bootstrap.cc:654] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.481566 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ec01cade6e41d1a0ff6c8929ee8155" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" is_pre_election: true
I20240502 08:18:11.481873 19127 raft_consensus.cc:2454] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 0.
I20240502 08:18:11.482789 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48005466a39437b937e2d09c1826d27" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" is_pre_election: true
I20240502 08:18:11.483065 19127 raft_consensus.cc:2454] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 0.
I20240502 08:18:11.483728 19366 raft_consensus.cc:482] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.483970 19366 raft_consensus.cc:504] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.485064 19366 leader_election.cc:288] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.485836 19242 leader_election.cc:302] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:11.486059 19204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ec01cade6e41d1a0ff6c8929ee8155" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60" is_pre_election: true
I20240502 08:18:11.486249 19242 leader_election.cc:302] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:11.486320 19204 raft_consensus.cc:2454] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 0.
I20240502 08:18:11.487002 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa05fba592744fdbfb354d227626c26" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" is_pre_election: true
I20240502 08:18:11.487267 19127 raft_consensus.cc:2454] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 0.
I20240502 08:18:11.487463 19204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48005466a39437b937e2d09c1826d27" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60" is_pre_election: true
I20240502 08:18:11.487715 19367 raft_consensus.cc:2781] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.487768 19204 raft_consensus.cc:2454] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 0.
I20240502 08:18:11.487862 19367 raft_consensus.cc:482] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.487985 19367 raft_consensus.cc:3037] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.488871 19204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa05fba592744fdbfb354d227626c26" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60" is_pre_election: true
I20240502 08:18:11.489140 19204 raft_consensus.cc:2454] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 0.
I20240502 08:18:11.490166 19243 leader_election.cc:302] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 7d576eb536a94138b805dc5ef8b0800c; no voters:
I20240502 08:18:11.490621 19330 raft_consensus.cc:2781] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.490794 19330 raft_consensus.cc:482] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.490926 19330 raft_consensus.cc:3037] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.492684 19331 raft_consensus.cc:482] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.492844 19331 raft_consensus.cc:504] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.493429 19325 tablet_bootstrap.cc:492] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60: No bootstrap required, opened a new log
I20240502 08:18:11.493736 19325 ts_tablet_manager.cc:1367] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.004s
I20240502 08:18:11.493894 19331 leader_election.cc:288] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:11.494408 19330 raft_consensus.cc:504] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.495182 19330 leader_election.cc:288] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.495098 19325 raft_consensus.cc:348] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.495405 19330 raft_consensus.cc:2781] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.495550 19325 raft_consensus.cc:374] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.495864 19325 raft_consensus.cc:729] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.496620 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48005466a39437b937e2d09c1826d27" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:11.497714 19127 raft_consensus.cc:3037] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.499274 19367 raft_consensus.cc:504] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.496852 19331 raft_consensus.cc:482] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.500192 19331 raft_consensus.cc:504] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.500854 19331 leader_election.cc:288] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:11.497418 19204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48005466a39437b937e2d09c1826d27" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:11.500886 19367 leader_election.cc:288] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.499709 19325 consensus_queue.cc:260] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.501099 19126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ec01cade6e41d1a0ff6c8929ee8155" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:11.499974 19365 raft_consensus.cc:482] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.501379 19126 raft_consensus.cc:3037] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.502655 19325 ts_tablet_manager.cc:1398] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:18:11.503213 19326 tablet_bootstrap.cc:492] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154: No bootstrap required, opened a new log
I20240502 08:18:11.503414 19326 ts_tablet_manager.cc:1367] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.010s
I20240502 08:18:11.503690 19204 raft_consensus.cc:3037] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.504258 19326 raft_consensus.cc:348] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.504491 19326 raft_consensus.cc:374] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.504576 19326 raft_consensus.cc:729] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.504812 19127 raft_consensus.cc:2454] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 1.
I20240502 08:18:11.504904 19326 consensus_queue.cc:260] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.501425 19365 raft_consensus.cc:504] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.505859 19332 raft_consensus.cc:482] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.506019 19332 raft_consensus.cc:504] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.506323 19365 leader_election.cc:288] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.506657 19204 raft_consensus.cc:2454] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 1.
I20240502 08:18:11.495721 19330 raft_consensus.cc:482] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.506870 19330 raft_consensus.cc:3037] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.506949 19202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4febdd8875b64cc1a156527854a1167a" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60" is_pre_election: true
I20240502 08:18:11.507031 19203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ec01cade6e41d1a0ff6c8929ee8155" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:11.507195 19332 leader_election.cc:288] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:11.507252 19202 raft_consensus.cc:2454] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 0.
I20240502 08:18:11.507318 19203 raft_consensus.cc:3037] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.508742 19324 tablet_bootstrap.cc:492] T 5990222a54794361819bebe9e51582e5 P 7d576eb536a94138b805dc5ef8b0800c: No bootstrap required, opened a new log
I20240502 08:18:11.508988 19324 ts_tablet_manager.cc:1367] T 5990222a54794361819bebe9e51582e5 P 7d576eb536a94138b805dc5ef8b0800c: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.003s
I20240502 08:18:11.509994 19330 raft_consensus.cc:504] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.509955 19324 raft_consensus.cc:348] T 5990222a54794361819bebe9e51582e5 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.510636 19330 leader_election.cc:288] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.511461 19203 raft_consensus.cc:2454] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 1.
I20240502 08:18:11.514279 19242 leader_election.cc:302] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:11.515345 19330 raft_consensus.cc:482] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.515491 19330 raft_consensus.cc:504] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.516114 19330 leader_election.cc:288] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.501390 19023 catalog_manager.cc:5484] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3). New cstate: current_term: 1 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:11.516270 19330 raft_consensus.cc:2781] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.516372 19330 raft_consensus.cc:686] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Becoming Leader. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:11.516606 19330 consensus_queue.cc:237] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.518826 19279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e844ad43672042da8eabbc9b74e622f7" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d576eb536a94138b805dc5ef8b0800c" is_pre_election: true
I20240502 08:18:11.518954 19278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db356e6346d948938202e72df77e8199" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d576eb536a94138b805dc5ef8b0800c" is_pre_election: true
I20240502 08:18:11.519052 19279 raft_consensus.cc:2454] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 0.
I20240502 08:18:11.519100 19278 raft_consensus.cc:2454] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 0.
I20240502 08:18:11.519454 19166 leader_election.cc:302] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 7d576eb536a94138b805dc5ef8b0800c; no voters:
I20240502 08:18:11.519767 19331 raft_consensus.cc:2781] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.519858 19331 raft_consensus.cc:482] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.519937 19331 raft_consensus.cc:3037] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.519989 19166 leader_election.cc:302] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 7d576eb536a94138b805dc5ef8b0800c; no voters:
I20240502 08:18:11.522115 19126 raft_consensus.cc:2454] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 1.
I20240502 08:18:11.523010 19331 raft_consensus.cc:504] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.523872 19331 leader_election.cc:288] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 1 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:11.524034 19331 raft_consensus.cc:2781] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.524101 19331 raft_consensus.cc:482] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.524185 19331 raft_consensus.cc:3037] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.525668 19324 raft_consensus.cc:374] T 5990222a54794361819bebe9e51582e5 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.525830 19324 raft_consensus.cc:729] T 5990222a54794361819bebe9e51582e5 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.526139 19324 consensus_queue.cc:260] T 5990222a54794361819bebe9e51582e5 P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.526942 19324 ts_tablet_manager.cc:1398] T 5990222a54794361819bebe9e51582e5 P 7d576eb536a94138b805dc5ef8b0800c: Time spent starting tablet: real 0.018s user 0.010s sys 0.005s
I20240502 08:18:11.527364 19243 leader_election.cc:302] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 7d576eb536a94138b805dc5ef8b0800c; no voters:
I20240502 08:18:11.527529 19202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db356e6346d948938202e72df77e8199" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60" is_pre_election: true
I20240502 08:18:11.527655 19243 leader_election.cc:302] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 7d576eb536a94138b805dc5ef8b0800c; no voters:
I20240502 08:18:11.527930 19365 raft_consensus.cc:2781] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.527956 19203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa05fba592744fdbfb354d227626c26" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:11.528003 19367 raft_consensus.cc:2781] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.528117 19203 raft_consensus.cc:3037] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.528148 19367 raft_consensus.cc:686] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Becoming Leader. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:11.528445 19367 consensus_queue.cc:237] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.528023 19365 raft_consensus.cc:482] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.530174 19365 raft_consensus.cc:3037] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.531759 19203 raft_consensus.cc:2454] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 1.
I20240502 08:18:11.532255 19243 leader_election.cc:302] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 7d576eb536a94138b805dc5ef8b0800c; no voters:
I20240502 08:18:11.532579 19366 raft_consensus.cc:2781] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.535256 19377 raft_consensus.cc:482] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.535413 19377 raft_consensus.cc:504] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.537467 19365 raft_consensus.cc:504] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.538400 19365 leader_election.cc:288] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.539147 19365 raft_consensus.cc:482] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.539505 19365 raft_consensus.cc:504] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.540354 19365 leader_election.cc:288] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.539206 19331 raft_consensus.cc:504] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.541199 19202 raft_consensus.cc:2379] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d576eb536a94138b805dc5ef8b0800c in current term 1: Already voted for candidate 287e8970c9524c998a437fbb3349fb60 in this term.
I20240502 08:18:11.542872 19331 leader_election.cc:288] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 1 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:11.535653 19126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4febdd8875b64cc1a156527854a1167a" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" is_pre_election: true
I20240502 08:18:11.547859 19126 raft_consensus.cc:2454] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 0.
I20240502 08:18:11.549461 19278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e844ad43672042da8eabbc9b74e622f7" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
I20240502 08:18:11.549695 19278 raft_consensus.cc:3037] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.549790 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db356e6346d948938202e72df77e8199" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" is_pre_election: true
I20240502 08:18:11.549998 19127 raft_consensus.cc:2454] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 0.
I20240502 08:18:11.550554 19242 leader_election.cc:302] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:11.550894 19330 raft_consensus.cc:2781] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.550981 19330 raft_consensus.cc:482] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.551059 19330 raft_consensus.cc:3037] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.552008 19377 leader_election.cc:288] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:11.552728 19278 raft_consensus.cc:2454] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 1.
I20240502 08:18:11.552784 19366 raft_consensus.cc:686] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Becoming Leader. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:11.553035 19125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4febdd8875b64cc1a156527854a1167a" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:11.553190 19125 raft_consensus.cc:3037] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.553093 19366 consensus_queue.cc:237] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.554214 19330 raft_consensus.cc:504] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.554983 19330 leader_election.cc:288] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.556053 19125 raft_consensus.cc:2454] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 1.
I20240502 08:18:11.556344 19124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7253eb5fb84187805cff227d030f54" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" is_pre_election: true
I20240502 08:18:11.556531 19124 raft_consensus.cc:2454] T 7d7253eb5fb84187805cff227d030f54 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 0.
I20240502 08:18:11.549484 19126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa05fba592744fdbfb354d227626c26" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:11.557787 19126 raft_consensus.cc:3037] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.559413 19023 catalog_manager.cc:5484] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3). New cstate: current_term: 1 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:11.558496 19278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db356e6346d948938202e72df77e8199" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
I20240502 08:18:11.559965 19278 raft_consensus.cc:2379] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 287e8970c9524c998a437fbb3349fb60 in current term 1: Already voted for candidate 7d576eb536a94138b805dc5ef8b0800c in this term.
I20240502 08:18:11.560292 19279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7253eb5fb84187805cff227d030f54" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d576eb536a94138b805dc5ef8b0800c" is_pre_election: true
I20240502 08:18:11.560442 19279 raft_consensus.cc:2454] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 0.
I20240502 08:18:11.562044 19166 leader_election.cc:302] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 7d576eb536a94138b805dc5ef8b0800c; no voters:
I20240502 08:18:11.562384 19166 leader_election.cc:302] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 7d576eb536a94138b805dc5ef8b0800c; no voters:
I20240502 08:18:11.562852 19126 raft_consensus.cc:2454] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 1.
I20240502 08:18:11.563164 19242 leader_election.cc:302] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:11.563361 19331 raft_consensus.cc:2781] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.564673 19242 leader_election.cc:302] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:11.564980 19379 raft_consensus.cc:2781] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.565147 19379 raft_consensus.cc:686] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Becoming Leader. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:11.565446 19126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db356e6346d948938202e72df77e8199" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:11.566304 19126 raft_consensus.cc:3037] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.568873 19023 catalog_manager.cc:5484] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3). New cstate: current_term: 1 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:11.569355 19367 raft_consensus.cc:2781] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.569494 19377 raft_consensus.cc:2781] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.565482 19379 consensus_queue.cc:237] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.569492 19023 catalog_manager.cc:5484] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3). New cstate: current_term: 1 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:11.570572 19124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e844ad43672042da8eabbc9b74e622f7" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" is_pre_election: true
I20240502 08:18:11.570672 19126 raft_consensus.cc:2454] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 1.
I20240502 08:18:11.570773 19124 raft_consensus.cc:2454] T e844ad43672042da8eabbc9b74e622f7 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 0.
I20240502 08:18:11.571187 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e844ad43672042da8eabbc9b74e622f7" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:11.571440 19127 raft_consensus.cc:3037] T e844ad43672042da8eabbc9b74e622f7 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.571794 19123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db356e6346d948938202e72df77e8199" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:11.572233 19123 raft_consensus.cc:2379] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 287e8970c9524c998a437fbb3349fb60 in current term 1: Already voted for candidate 7d576eb536a94138b805dc5ef8b0800c in this term.
I20240502 08:18:11.569509 19367 raft_consensus.cc:482] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.518911 19326 ts_tablet_manager.cc:1398] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154: Time spent starting tablet: real 0.015s user 0.000s sys 0.003s
I20240502 08:18:11.572772 19367 raft_consensus.cc:3037] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.572672 19122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7253eb5fb84187805cff227d030f54" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" is_pre_election: true
I20240502 08:18:11.573002 19122 raft_consensus.cc:2454] T 7d7253eb5fb84187805cff227d030f54 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 0.
I20240502 08:18:11.575299 19367 raft_consensus.cc:504] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.575652 19377 raft_consensus.cc:482] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.575771 19377 raft_consensus.cc:3037] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.577313 19331 raft_consensus.cc:686] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 1 LEADER]: Becoming Leader. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Running, Role: LEADER
I20240502 08:18:11.577653 19331 consensus_queue.cc:237] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.578092 19127 raft_consensus.cc:2454] T e844ad43672042da8eabbc9b74e622f7 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 1.
I20240502 08:18:11.578974 19367 leader_election.cc:288] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.580569 19125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db356e6346d948938202e72df77e8199" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" is_pre_election: true
I20240502 08:18:11.580792 19125 raft_consensus.cc:2379] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 287e8970c9524c998a437fbb3349fb60 in current term 1: Already voted for candidate 7d576eb536a94138b805dc5ef8b0800c in this term.
I20240502 08:18:11.579624 19377 raft_consensus.cc:504] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.581308 19167 leader_election.cc:302] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60; no voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154
I20240502 08:18:11.581555 19203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f640265eb3f4b10a0675e3b08b11680" candidate_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60" is_pre_election: true
I20240502 08:18:11.581750 19203 raft_consensus.cc:2454] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86029a0bbf0b4f77b9c0e96cf121e154 in term 0.
I20240502 08:18:11.582127 19092 leader_election.cc:302] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:11.582187 19279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7253eb5fb84187805cff227d030f54" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
I20240502 08:18:11.582504 19279 raft_consensus.cc:2379] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 287e8970c9524c998a437fbb3349fb60 in current term 1: Already voted for candidate 7d576eb536a94138b805dc5ef8b0800c in this term.
I20240502 08:18:11.582580 19125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7253eb5fb84187805cff227d030f54" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:11.582819 19125 raft_consensus.cc:3037] T 7d7253eb5fb84187805cff227d030f54 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.583518 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7253eb5fb84187805cff227d030f54" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:11.584607 19242 leader_election.cc:302] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:11.584981 19202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7253eb5fb84187805cff227d030f54" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60" is_pre_election: true
I20240502 08:18:11.585211 19202 raft_consensus.cc:2379] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d576eb536a94138b805dc5ef8b0800c in current term 1: Already voted for candidate 287e8970c9524c998a437fbb3349fb60 in this term.
I20240502 08:18:11.585270 19204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db356e6346d948938202e72df77e8199" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:11.585865 19279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f640265eb3f4b10a0675e3b08b11680" candidate_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d576eb536a94138b805dc5ef8b0800c" is_pre_election: true
I20240502 08:18:11.585999 19381 raft_consensus.cc:2781] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.586069 19279 raft_consensus.cc:2454] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86029a0bbf0b4f77b9c0e96cf121e154 in term 0.
I20240502 08:18:11.586138 19381 raft_consensus.cc:686] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Becoming Leader. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:11.586292 19379 raft_consensus.cc:482] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.586457 19379 raft_consensus.cc:504] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.586439 19381 consensus_queue.cc:237] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.586822 19203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4febdd8875b64cc1a156527854a1167a" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:11.586985 19203 raft_consensus.cc:3037] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.587271 19379 leader_election.cc:288] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.587420 19379 raft_consensus.cc:482] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.587563 19379 raft_consensus.cc:504] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.588187 19379 leader_election.cc:288] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.588714 19377 leader_election.cc:288] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 1 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:11.588797 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fa1524bb0a749f9998928d10f4adea6" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" is_pre_election: true
I20240502 08:18:11.589073 19122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f640265eb3f4b10a0675e3b08b11680" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" is_pre_election: true
I20240502 08:18:11.589241 19122 raft_consensus.cc:2454] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 0.
I20240502 08:18:11.589468 19127 raft_consensus.cc:2454] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 0.
I20240502 08:18:11.589738 19242 leader_election.cc:302] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:11.589999 19203 raft_consensus.cc:2454] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 1.
I20240502 08:18:11.590056 19330 raft_consensus.cc:2781] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.590159 19330 raft_consensus.cc:482] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.590215 19242 leader_election.cc:302] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:11.590281 19330 raft_consensus.cc:3037] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.590458 19367 raft_consensus.cc:2781] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.590652 19367 raft_consensus.cc:482] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.590796 19367 raft_consensus.cc:3037] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.592700 19024 catalog_manager.cc:5484] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3). New cstate: current_term: 1 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:11.593235 19330 raft_consensus.cc:504] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.593817 19367 raft_consensus.cc:504] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.594285 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f640265eb3f4b10a0675e3b08b11680" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:11.594494 19127 raft_consensus.cc:3037] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.594687 19367 leader_election.cc:288] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.594846 19023 catalog_manager.cc:5484] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 287e8970c9524c998a437fbb3349fb60 (127.15.81.2). New cstate: current_term: 1 leader_uuid: "287e8970c9524c998a437fbb3349fb60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:11.595762 19330 leader_election.cc:288] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:11.595803 19122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fa1524bb0a749f9998928d10f4adea6" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:11.595968 19122 raft_consensus.cc:3037] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.596542 19202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fa1524bb0a749f9998928d10f4adea6" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60" is_pre_election: true
I20240502 08:18:11.596753 19202 raft_consensus.cc:2454] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 0.
I20240502 08:18:11.597064 19201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f640265eb3f4b10a0675e3b08b11680" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60" is_pre_election: true
I20240502 08:18:11.597343 19201 raft_consensus.cc:2454] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 0.
I20240502 08:18:11.597673 19200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f640265eb3f4b10a0675e3b08b11680" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:11.597923 19200 raft_consensus.cc:3037] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.598351 19127 raft_consensus.cc:2454] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 1.
I20240502 08:18:11.599794 19202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fa1524bb0a749f9998928d10f4adea6" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:11.599983 19202 raft_consensus.cc:3037] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.600047 19242 leader_election.cc:302] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:11.600139 19377 raft_consensus.cc:482] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.600345 19377 raft_consensus.cc:504] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.600526 19330 raft_consensus.cc:2781] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.600692 19330 raft_consensus.cc:686] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Becoming Leader. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:11.600956 19330 consensus_queue.cc:237] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.600044 19200 raft_consensus.cc:2454] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 1.
I20240502 08:18:11.601419 19279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3591d08cc604b349553a185225ef943" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d576eb536a94138b805dc5ef8b0800c" is_pre_election: true
I20240502 08:18:11.601683 19279 raft_consensus.cc:2454] T d3591d08cc604b349553a185225ef943 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 0.
I20240502 08:18:11.601400 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3591d08cc604b349553a185225ef943" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" is_pre_election: true
I20240502 08:18:11.599594 19122 raft_consensus.cc:2454] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 1.
I20240502 08:18:11.602104 19166 leader_election.cc:302] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 7d576eb536a94138b805dc5ef8b0800c; no voters:
I20240502 08:18:11.602242 19242 leader_election.cc:302] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:11.602355 19127 raft_consensus.cc:2454] T d3591d08cc604b349553a185225ef943 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 0.
I20240502 08:18:11.602396 19331 raft_consensus.cc:2781] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.602532 19367 raft_consensus.cc:2781] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.602648 19331 raft_consensus.cc:482] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.602743 19367 raft_consensus.cc:686] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Becoming Leader. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:11.602846 19377 leader_election.cc:288] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:11.602864 19331 raft_consensus.cc:3037] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.603030 19367 consensus_queue.cc:237] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.603233 19202 raft_consensus.cc:2454] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 1.
I20240502 08:18:11.604509 19125 raft_consensus.cc:2454] T 7d7253eb5fb84187805cff227d030f54 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 1.
I20240502 08:18:11.604889 19167 leader_election.cc:302] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 86029a0bbf0b4f77b9c0e96cf121e154; no voters: 7d576eb536a94138b805dc5ef8b0800c
I20240502 08:18:11.605234 19377 raft_consensus.cc:2781] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.605707 19331 raft_consensus.cc:504] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.606495 19331 leader_election.cc:288] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 1 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:11.606990 19279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3591d08cc604b349553a185225ef943" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
I20240502 08:18:11.607218 19279 raft_consensus.cc:3037] T d3591d08cc604b349553a185225ef943 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.607364 19203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7253eb5fb84187805cff227d030f54" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:11.607823 19332 raft_consensus.cc:2741] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240502 08:18:11.608156 19125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3591d08cc604b349553a185225ef943" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:11.608361 19125 raft_consensus.cc:3037] T d3591d08cc604b349553a185225ef943 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.610934 19279 raft_consensus.cc:2454] T d3591d08cc604b349553a185225ef943 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 1.
I20240502 08:18:11.612043 19332 raft_consensus.cc:482] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.612242 19332 raft_consensus.cc:504] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.612515 19166 leader_election.cc:302] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 7d576eb536a94138b805dc5ef8b0800c; no voters:
I20240502 08:18:11.613018 19332 leader_election.cc:288] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:11.613618 19279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5990222a54794361819bebe9e51582e5" candidate_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d576eb536a94138b805dc5ef8b0800c" is_pre_election: true
I20240502 08:18:11.613919 19279 raft_consensus.cc:2454] T 5990222a54794361819bebe9e51582e5 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86029a0bbf0b4f77b9c0e96cf121e154 in term 0.
I20240502 08:18:11.614184 19202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5990222a54794361819bebe9e51582e5" candidate_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60" is_pre_election: true
I20240502 08:18:11.614404 19090 leader_election.cc:302] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:11.614478 19202 raft_consensus.cc:2454] T 5990222a54794361819bebe9e51582e5 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86029a0bbf0b4f77b9c0e96cf121e154 in term 0.
I20240502 08:18:11.614773 19332 raft_consensus.cc:2781] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.615008 19332 raft_consensus.cc:482] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.615144 19332 raft_consensus.cc:3037] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.615226 19125 raft_consensus.cc:2454] T d3591d08cc604b349553a185225ef943 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 1.
I20240502 08:18:11.581564 19383 raft_consensus.cc:2726] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:18:11.616268 19204 raft_consensus.cc:2379] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d576eb536a94138b805dc5ef8b0800c in current term 1: Already voted for candidate 287e8970c9524c998a437fbb3349fb60 in this term.
I20240502 08:18:11.613595 19331 raft_consensus.cc:2781] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.617503 19331 raft_consensus.cc:686] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [term 1 LEADER]: Becoming Leader. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Running, Role: LEADER
I20240502 08:18:11.618075 19331 consensus_queue.cc:237] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.618800 19332 raft_consensus.cc:504] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.619791 19204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5990222a54794361819bebe9e51582e5" candidate_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:11.619992 19204 raft_consensus.cc:3037] T 5990222a54794361819bebe9e51582e5 P 287e8970c9524c998a437fbb3349fb60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.619958 19023 catalog_manager.cc:5484] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3). New cstate: current_term: 1 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:11.620738 19377 raft_consensus.cc:686] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [term 1 LEADER]: Becoming Leader. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Running, Role: LEADER
I20240502 08:18:11.621186 19377 consensus_queue.cc:237] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.622721 19204 raft_consensus.cc:2454] T 5990222a54794361819bebe9e51582e5 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86029a0bbf0b4f77b9c0e96cf121e154 in term 1.
I20240502 08:18:11.623248 19092 leader_election.cc:302] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:11.628800 19279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5990222a54794361819bebe9e51582e5" candidate_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
I20240502 08:18:11.629141 19279 raft_consensus.cc:3037] T 5990222a54794361819bebe9e51582e5 P 7d576eb536a94138b805dc5ef8b0800c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.632836 19023 catalog_manager.cc:5484] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 287e8970c9524c998a437fbb3349fb60 (127.15.81.2). New cstate: current_term: 1 leader_uuid: "287e8970c9524c998a437fbb3349fb60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:11.633971 19243 leader_election.cc:302] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c; no voters: 287e8970c9524c998a437fbb3349fb60, 86029a0bbf0b4f77b9c0e96cf121e154
I20240502 08:18:11.634279 19381 raft_consensus.cc:2726] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:18:11.636287 19279 raft_consensus.cc:2454] T 5990222a54794361819bebe9e51582e5 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86029a0bbf0b4f77b9c0e96cf121e154 in term 1.
I20240502 08:18:11.637226 19332 leader_election.cc:288] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [CANDIDATE]: Term 1 election: Requested vote from peers 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:11.637493 19332 raft_consensus.cc:2781] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.637646 19332 raft_consensus.cc:686] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 LEADER]: Becoming Leader. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Running, Role: LEADER
I20240502 08:18:11.638676 19332 consensus_queue.cc:237] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:11.638664 19023 catalog_manager.cc:5484] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 287e8970c9524c998a437fbb3349fb60 (127.15.81.2). New cstate: current_term: 1 leader_uuid: "287e8970c9524c998a437fbb3349fb60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:11.643484 19024 catalog_manager.cc:5484] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3). New cstate: current_term: 1 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:11.644402 19024 catalog_manager.cc:5484] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3). New cstate: current_term: 1 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:11.649163 19023 catalog_manager.cc:5484] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1). New cstate: current_term: 1 leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:11.828603 19203 raft_consensus.cc:1261] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.829728 19330 consensus_queue.cc:1035] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.839915 19125 raft_consensus.cc:1261] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.839924 19204 raft_consensus.cc:1261] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.840605 19330 consensus_queue.cc:1035] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.840926 19381 consensus_queue.cc:1035] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.855336 19127 raft_consensus.cc:1261] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.855581 19200 raft_consensus.cc:1261] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.856122 19381 consensus_queue.cc:1035] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.855336 19202 raft_consensus.cc:1261] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.858829 19330 consensus_queue.cc:1035] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.861606 19381 consensus_queue.cc:1035] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.863035 19122 raft_consensus.cc:1261] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.864614 19122 raft_consensus.cc:1261] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.865355 19367 consensus_queue.cc:1035] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.866156 19367 consensus_queue.cc:1035] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.867313 19201 raft_consensus.cc:1261] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.867911 19078 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:18:11.870961 19279 raft_consensus.cc:1261] T 5990222a54794361819bebe9e51582e5 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.871735 19332 consensus_queue.cc:1035] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.873378 19201 raft_consensus.cc:1261] T 5990222a54794361819bebe9e51582e5 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.873980 19392 consensus_queue.cc:1035] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.875773 19381 consensus_queue.cc:1035] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.875775 19199 raft_consensus.cc:1261] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.880254 19278 raft_consensus.cc:1261] T 7d7253eb5fb84187805cff227d030f54 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.881223 19331 consensus_queue.cc:1035] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.881567 19330 consensus_queue.cc:1035] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.881824 19124 raft_consensus.cc:1261] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.882390 19330 consensus_queue.cc:1035] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.895243 19126 raft_consensus.cc:1261] T e844ad43672042da8eabbc9b74e622f7 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.895298 19277 raft_consensus.cc:1261] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.896339 19276 raft_consensus.cc:1261] T d3591d08cc604b349553a185225ef943 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.896555 19126 raft_consensus.cc:1261] T 7d7253eb5fb84187805cff227d030f54 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.897249 19331 consensus_queue.cc:1035] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.897450 19389 consensus_queue.cc:1035] T 7d7253eb5fb84187805cff227d030f54 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.897852 19331 consensus_queue.cc:1035] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.898162 19383 consensus_queue.cc:1035] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.899415 19121 raft_consensus.cc:1261] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.900154 19330 consensus_queue.cc:1035] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.902637 19197 raft_consensus.cc:1261] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.903380 19381 consensus_queue.cc:1035] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.925369 19119 raft_consensus.cc:1261] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.926090 19196 raft_consensus.cc:1261] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.926661 19381 consensus_queue.cc:1035] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.926934 19330 consensus_queue.cc:1035] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.935060 19118 raft_consensus.cc:1261] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.935886 19381 consensus_queue.cc:1035] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.940714 19419 mvcc.cc:205] Tried to move back new op lower bound from 7023156805088931840 to 7023156804155191296. Current Snapshot: MvccSnapshot[applied={T|T < 7023156805088931840}]
I20240502 08:18:12.016502 19119 raft_consensus.cc:1261] T d3591d08cc604b349553a185225ef943 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:12.017680 19331 consensus_queue.cc:1035] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.050652 19428 mvcc.cc:205] Tried to move back new op lower bound from 7023156805209788416 to 7023156804008783872. Current Snapshot: MvccSnapshot[applied={T|T < 7023156805209788416}]
I20240502 08:18:12.245111 19417 mvcc.cc:205] Tried to move back new op lower bound from 7023156805023350784 to 7023156803703668736. Current Snapshot: MvccSnapshot[applied={T|T < 7023156805023350784}]
I20240502 08:18:12.298345 19276 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f640265eb3f4b10a0675e3b08b11680"
dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
mode: GRACEFUL
new_leader_uuid: "287e8970c9524c998a437fbb3349fb60"
from {username='slave'} at 127.0.0.1:52856
I20240502 08:18:12.319777 19276 raft_consensus.cc:595] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Received request to transfer leadership to TS 287e8970c9524c998a437fbb3349fb60
I20240502 08:18:12.346381 19381 raft_consensus.cc:979] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c: : Instructing follower 287e8970c9524c998a437fbb3349fb60 to start an election
I20240502 08:18:12.351037 19276 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4febdd8875b64cc1a156527854a1167a"
dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
mode: GRACEFUL
new_leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
from {username='slave'} at 127.0.0.1:52856
I20240502 08:18:12.356606 19460 raft_consensus.cc:1067] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Signalling peer 287e8970c9524c998a437fbb3349fb60 to start an election
I20240502 08:18:12.356803 19276 raft_consensus.cc:595] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Received request to transfer leadership to TS 86029a0bbf0b4f77b9c0e96cf121e154
W20240502 08:18:12.364185 19258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.371701 19255 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.378589 19255 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
I20240502 08:18:12.385100 19275 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5fa05fba592744fdbfb354d227626c26"
dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
mode: GRACEFUL
new_leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
from {username='slave'} at 127.0.0.1:52856
I20240502 08:18:12.398141 19275 raft_consensus.cc:595] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Received request to transfer leadership to TS 86029a0bbf0b4f77b9c0e96cf121e154
I20240502 08:18:12.398941 19367 raft_consensus.cc:979] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c: : Instructing follower 86029a0bbf0b4f77b9c0e96cf121e154 to start an election
I20240502 08:18:12.398591 19197 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2f640265eb3f4b10a0675e3b08b11680"
dest_uuid: "287e8970c9524c998a437fbb3349fb60"
from {username='slave'} at 127.0.0.1:53040
I20240502 08:18:12.410640 19367 raft_consensus.cc:1067] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Signalling peer 86029a0bbf0b4f77b9c0e96cf121e154 to start an election
I20240502 08:18:12.417665 19197 raft_consensus.cc:482] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:12.417917 19197 raft_consensus.cc:3037] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:12.419883 19255 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
I20240502 08:18:12.423274 19125 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4febdd8875b64cc1a156527854a1167a"
dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
from {username='slave'} at 127.0.0.1:33030
I20240502 08:18:12.423516 19125 raft_consensus.cc:482] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:12.430181 19125 raft_consensus.cc:3037] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:12.432371 19276 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b48005466a39437b937e2d09c1826d27"
dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
mode: GRACEFUL
new_leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
from {username='slave'} at 127.0.0.1:52856
W20240502 08:18:12.436739 19258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.436888 19255 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.445384 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
I20240502 08:18:12.447106 19276 raft_consensus.cc:595] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Received request to transfer leadership to TS 86029a0bbf0b4f77b9c0e96cf121e154
I20240502 08:18:12.449131 19078 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240502 08:18:12.447928 19197 raft_consensus.cc:504] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:12.451202 19078 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:18:12.452883 19197 leader_election.cc:288] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 2 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
W20240502 08:18:12.448876 19243 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.81.2:38313, user_credentials={real_user=slave}} blocked reactor thread for 43822.3us
W20240502 08:18:12.454545 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
I20240502 08:18:12.457366 19275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f640265eb3f4b10a0675e3b08b11680" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
I20240502 08:18:12.459983 19275 raft_consensus.cc:3032] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:12.462304 19125 raft_consensus.cc:504] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
W20240502 08:18:12.460896 19403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
I20240502 08:18:12.466356 19120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f640265eb3f4b10a0675e3b08b11680" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:12.467759 19125 leader_election.cc:288] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [CANDIDATE]: Term 2 election: Requested vote from peers 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:12.467805 19120 raft_consensus.cc:3037] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:12.471371 19275 raft_consensus.cc:729] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:12.475800 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4febdd8875b64cc1a156527854a1167a" candidate_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:12.476049 19198 raft_consensus.cc:3037] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:12.478507 19276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4febdd8875b64cc1a156527854a1167a" candidate_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
I20240502 08:18:12.476401 19275 consensus_queue.cc:260] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:12.485396 19275 raft_consensus.cc:3037] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:12.487273 19257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Illegal state: replica 7d576eb536a94138b805dc5ef8b0800c is not leader of this config: current role FOLLOWER
I20240502 08:18:12.487867 19276 raft_consensus.cc:3032] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Stepping down as leader of term 1
W20240502 08:18:12.489114 19255 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.489981 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Illegal state: replica 7d576eb536a94138b805dc5ef8b0800c is not leader of this config: current role FOLLOWER
W20240502 08:18:12.490217 19257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Illegal state: replica 7d576eb536a94138b805dc5ef8b0800c is not leader of this config: current role FOLLOWER
I20240502 08:18:12.490586 19276 raft_consensus.cc:729] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
W20240502 08:18:12.491370 19258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Illegal state: replica 7d576eb536a94138b805dc5ef8b0800c is not leader of this config: current role FOLLOWER
W20240502 08:18:12.493438 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.493395 19258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.493803 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
I20240502 08:18:12.493391 19198 raft_consensus.cc:2454] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86029a0bbf0b4f77b9c0e96cf121e154 in term 2.
I20240502 08:18:12.494386 19092 leader_election.cc:302] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:12.494777 19392 raft_consensus.cc:2781] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:12.495396 19392 raft_consensus.cc:686] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Becoming Leader. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Running, Role: LEADER
I20240502 08:18:12.495848 19392 consensus_queue.cc:237] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
W20240502 08:18:12.498319 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.500190 19106 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33042: Illegal state: replica 86029a0bbf0b4f77b9c0e96cf121e154 is not leader of this config: current role FOLLOWER
I20240502 08:18:12.497148 19276 consensus_queue.cc:260] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:12.504115 19276 raft_consensus.cc:3037] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:12.504904 19106 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33042: Illegal state: replica 86029a0bbf0b4f77b9c0e96cf121e154 is not leader of this config: current role FOLLOWER
I20240502 08:18:12.505546 19120 raft_consensus.cc:2454] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 2.
W20240502 08:18:12.506461 19107 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33042: Illegal state: replica 86029a0bbf0b4f77b9c0e96cf121e154 is not leader of this config: current role FOLLOWER
W20240502 08:18:12.507136 19105 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33042: Illegal state: replica 86029a0bbf0b4f77b9c0e96cf121e154 is not leader of this config: current role FOLLOWER
I20240502 08:18:12.507985 19275 raft_consensus.cc:2454] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 2.
W20240502 08:18:12.508666 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Illegal state: replica 7d576eb536a94138b805dc5ef8b0800c is not leader of this config: current role FOLLOWER
I20240502 08:18:12.507139 19018 catalog_manager.cc:5484] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 reported cstate change: term changed from 1 to 2, leader changed from 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3) to 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1). New cstate: current_term: 2 leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:12.509655 19181 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53066: Illegal state: replica 287e8970c9524c998a437fbb3349fb60 is not leader of this config: current role FOLLOWER
W20240502 08:18:12.510316 19257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.510511 19258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
I20240502 08:18:12.510915 19167 leader_election.cc:302] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
W20240502 08:18:12.511473 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Illegal state: replica 7d576eb536a94138b805dc5ef8b0800c is not leader of this config: current role FOLLOWER
I20240502 08:18:12.512073 19470 raft_consensus.cc:2781] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Leader election won for term 2
W20240502 08:18:12.512187 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
I20240502 08:18:12.512449 19470 raft_consensus.cc:686] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 2 LEADER]: Becoming Leader. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Running, Role: LEADER
W20240502 08:18:12.514518 19184 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53066: Service unavailable: leader is not yet ready
I20240502 08:18:12.514878 19470 consensus_queue.cc:237] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
W20240502 08:18:12.515710 19184 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53066: Service unavailable: leader is not yet ready
W20240502 08:18:12.515908 19181 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53066: Service unavailable: leader is not yet ready
I20240502 08:18:12.519044 19276 raft_consensus.cc:2454] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86029a0bbf0b4f77b9c0e96cf121e154 in term 2.
I20240502 08:18:12.523214 19198 raft_consensus.cc:1261] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:12.522853 19020 catalog_manager.cc:5484] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 reported cstate change: term changed from 1 to 2, leader changed from 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3) to 287e8970c9524c998a437fbb3349fb60 (127.15.81.2). New cstate: current_term: 2 leader_uuid: "287e8970c9524c998a437fbb3349fb60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:12.524565 19462 consensus_queue.cc:1035] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:18:12.526906 19258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.527444 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.529141 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
I20240502 08:18:12.530787 19120 raft_consensus.cc:1261] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:18:12.532816 19470 consensus_queue.cc:1035] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:12.534258 19276 raft_consensus.cc:1261] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:18:12.534904 19470 consensus_queue.cc:1035] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:12.535773 19275 raft_consensus.cc:1261] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:12.537357 19332 consensus_queue.cc:1035] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:18:12.538252 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.545079 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.546746 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.547819 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.615885 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.618363 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.619364 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.620349 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.620494 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.633970 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.637559 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.649528 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.653252 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.653354 19258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.654274 19258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.656292 19258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.656422 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.657387 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.660039 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.661161 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.662286 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.663415 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.663571 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.667141 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.668789 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.669034 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.669063 19258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.670109 19258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.670251 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.677278 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.677335 19258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.677472 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.677537 19257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.678889 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.681313 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.682482 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.685999 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.687276 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.687395 19257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.688560 19257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.690954 19257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.696501 19257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.696571 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.697963 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.698127 19257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.698259 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.698495 19258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.704387 19258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.710551 19258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.710664 19257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.710667 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.711400 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.711681 19257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.711681 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.720413 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.721678 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.723085 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.723085 19257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.724411 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.727075 19257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.727068 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.734686 19257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.736179 19257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.736338 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.736618 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.736806 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.744755 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.744755 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.746120 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.748567 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.748589 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.751055 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.755421 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.761240 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.761279 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.762356 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.763727 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.763759 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.764902 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.775902 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.775946 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.777010 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.779729 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.779855 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.780926 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.782075 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.795302 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.795403 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.795403 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.798516 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.798592 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.798516 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.803452 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.813503 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.814836 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.815428 19254 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.815690 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.816216 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.817320 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.827117 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.833209 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.833292 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.835855 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.835883 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.837098 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.840444 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.851052 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.853442 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.854574 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.855785 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.856798 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.860121 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.862277 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.871865 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.873008 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.875344 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.875416 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.879032 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.884588 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.885779 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
I20240502 08:18:12.892491 19460 raft_consensus.cc:979] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c: : Instructing follower 86029a0bbf0b4f77b9c0e96cf121e154 to start an election
I20240502 08:18:12.892685 19460 raft_consensus.cc:1067] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Signalling peer 86029a0bbf0b4f77b9c0e96cf121e154 to start an election
I20240502 08:18:12.893401 19126 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b48005466a39437b937e2d09c1826d27"
dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
from {username='slave'} at 127.0.0.1:33030
W20240502 08:18:12.893429 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
I20240502 08:18:12.893635 19126 raft_consensus.cc:482] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:12.893783 19126 raft_consensus.cc:3037] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:12.894804 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
I20240502 08:18:12.896051 19126 raft_consensus.cc:504] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:12.896752 19126 leader_election.cc:288] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [CANDIDATE]: Term 2 election: Requested vote from peers 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:12.897148 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48005466a39437b937e2d09c1826d27" candidate_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "287e8970c9524c998a437fbb3349fb60"
W20240502 08:18:12.897301 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
I20240502 08:18:12.897271 19276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48005466a39437b937e2d09c1826d27" candidate_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
I20240502 08:18:12.897368 19198 raft_consensus.cc:3037] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:12.897495 19276 raft_consensus.cc:3032] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:12.897629 19276 raft_consensus.cc:729] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:12.897917 19276 consensus_queue.cc:260] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:12.898310 19276 raft_consensus.cc:3037] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:12.898931 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.899139 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Illegal state: replica 7d576eb536a94138b805dc5ef8b0800c is not leader of this config: current role FOLLOWER
I20240502 08:18:12.900272 19198 raft_consensus.cc:2454] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86029a0bbf0b4f77b9c0e96cf121e154 in term 2.
I20240502 08:18:12.900664 19276 raft_consensus.cc:2454] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86029a0bbf0b4f77b9c0e96cf121e154 in term 2.
I20240502 08:18:12.900676 19092 leader_election.cc:302] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:12.900990 19332 raft_consensus.cc:2781] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:12.901136 19332 raft_consensus.cc:686] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Becoming Leader. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Running, Role: LEADER
I20240502 08:18:12.901396 19332 consensus_queue.cc:237] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:12.903535 19020 catalog_manager.cc:5484] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 reported cstate change: term changed from 1 to 2, leader changed from 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3) to 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1). New cstate: current_term: 2 leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:12.908936 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Illegal state: replica 7d576eb536a94138b805dc5ef8b0800c is not leader of this config: current role FOLLOWER
W20240502 08:18:12.909013 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.915488 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.916719 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Illegal state: replica 7d576eb536a94138b805dc5ef8b0800c is not leader of this config: current role FOLLOWER
W20240502 08:18:12.921787 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
I20240502 08:18:12.921937 19276 raft_consensus.cc:1261] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:12.921968 19198 raft_consensus.cc:1261] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:12.922494 19392 consensus_queue.cc:1035] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:12.922782 19332 consensus_queue.cc:1035] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 08:18:12.924422 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.934329 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.939657 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.950655 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.950655 19256 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.959472 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.966162 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.974979 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
W20240502 08:18:12.976148 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Service unavailable: leader transfer in progress
I20240502 08:18:12.977025 19365 raft_consensus.cc:979] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c: : Instructing follower 86029a0bbf0b4f77b9c0e96cf121e154 to start an election
I20240502 08:18:12.977200 19365 raft_consensus.cc:1067] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Signalling peer 86029a0bbf0b4f77b9c0e96cf121e154 to start an election
I20240502 08:18:12.977721 19126 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5fa05fba592744fdbfb354d227626c26"
dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
from {username='slave'} at 127.0.0.1:33030
I20240502 08:18:12.977903 19126 raft_consensus.cc:482] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:12.978017 19126 raft_consensus.cc:3037] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:12.979979 19126 raft_consensus.cc:504] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:12.980561 19126 leader_election.cc:288] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [CANDIDATE]: Term 2 election: Requested vote from peers 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:12.981113 19276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa05fba592744fdbfb354d227626c26" candidate_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
I20240502 08:18:12.981119 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa05fba592744fdbfb354d227626c26" candidate_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:12.981389 19276 raft_consensus.cc:3032] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:12.981429 19198 raft_consensus.cc:3037] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:12.981473 19276 raft_consensus.cc:729] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:12.981734 19276 consensus_queue.cc:260] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:12.982025 19276 raft_consensus.cc:3037] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:12.983600 19198 raft_consensus.cc:2454] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86029a0bbf0b4f77b9c0e96cf121e154 in term 2.
I20240502 08:18:12.983978 19276 raft_consensus.cc:2454] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86029a0bbf0b4f77b9c0e96cf121e154 in term 2.
I20240502 08:18:12.984026 19092 leader_election.cc:302] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:12.984390 19462 raft_consensus.cc:2781] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:12.984613 19462 raft_consensus.cc:686] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Becoming Leader. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Running, Role: LEADER
I20240502 08:18:12.984901 19462 consensus_queue.cc:237] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
W20240502 08:18:12.987433 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Illegal state: replica 7d576eb536a94138b805dc5ef8b0800c is not leader of this config: current role FOLLOWER
I20240502 08:18:12.987457 19020 catalog_manager.cc:5484] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 reported cstate change: term changed from 1 to 2, leader changed from 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3) to 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1). New cstate: current_term: 2 leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:12.993135 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Illegal state: replica 7d576eb536a94138b805dc5ef8b0800c is not leader of this config: current role FOLLOWER
W20240502 08:18:12.998579 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Illegal state: replica 7d576eb536a94138b805dc5ef8b0800c is not leader of this config: current role FOLLOWER
W20240502 08:18:13.002080 19253 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52842: Illegal state: replica 7d576eb536a94138b805dc5ef8b0800c is not leader of this config: current role FOLLOWER
I20240502 08:18:13.015780 19198 raft_consensus.cc:1261] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:13.015786 19276 raft_consensus.cc:1261] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:13.016374 19462 consensus_queue.cc:1035] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:13.016641 19392 consensus_queue.cc:1035] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:13.491838 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:13.496515 19495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:13.497300 19494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:13.498170 19497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:13.498803 15684 server_base.cc:1034] running on GCE node
I20240502 08:18:13.499043 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:18:13.499100 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:18:13.499138 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637893499134 us; error 0 us; skew 500 ppm
I20240502 08:18:13.499275 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:13.500748 15684 webserver.cc:460] Webserver started at http://127.15.81.4:44233/ using document root <none> and password file <none>
I20240502 08:18:13.501004 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:13.501072 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:13.501221 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:13.501849 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-3-root/instance:
uuid: "4fa05fa6be33496798a23398a84ffbc9"
format_stamp: "Formatted at 2024-05-02 08:18:13 on dist-test-slave-rhmj"
I20240502 08:18:13.504391 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:18:13.505957 19502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:13.506347 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 08:18:13.506443 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-3-root
uuid: "4fa05fa6be33496798a23398a84ffbc9"
format_stamp: "Formatted at 2024-05-02 08:18:13 on dist-test-slave-rhmj"
I20240502 08:18:13.506532 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:13.515211 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:18:13.515748 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:13.516222 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:13.517046 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:13.517095 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:13.517163 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:13.517202 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:13.532500 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.4:39603
I20240502 08:18:13.532512 19565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.4:39603 every 8 connection(s)
I20240502 08:18:13.534276 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:13.537770 19570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:13.538483 19571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:13.541921 19573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:13.542640 15684 server_base.cc:1034] running on GCE node
I20240502 08:18:13.542874 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:18:13.542925 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:18:13.542950 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637893542946 us; error 0 us; skew 500 ppm
I20240502 08:18:13.543088 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:13.544713 15684 webserver.cc:460] Webserver started at http://127.15.81.5:39645/ using document root <none> and password file <none>
I20240502 08:18:13.544976 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:13.545063 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:13.545181 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:13.545781 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-4-root/instance:
uuid: "e6313d7fc95e46e88fb241f192368b1e"
format_stamp: "Formatted at 2024-05-02 08:18:13 on dist-test-slave-rhmj"
I20240502 08:18:13.549237 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 08:18:13.549777 19566 heartbeater.cc:347] Connected to a master server at 127.15.81.62:39387
I20240502 08:18:13.549904 19566 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:13.550328 19566 heartbeater.cc:511] Master 127.15.81.62:39387 requested a full tablet report, sending...
I20240502 08:18:13.551106 19578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:13.551403 19020 ts_manager.cc:194] Registered new tserver with Master: 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4:39603)
I20240502 08:18:13.551472 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240502 08:18:13.551569 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-4-root
uuid: "e6313d7fc95e46e88fb241f192368b1e"
format_stamp: "Formatted at 2024-05-02 08:18:13 on dist-test-slave-rhmj"
I20240502 08:18:13.551695 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:13.552520 19020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33142
I20240502 08:18:13.566907 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:18:13.567370 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:13.567922 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:13.568679 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:13.568732 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:13.568792 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:13.568836 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:13.585254 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.5:41865
I20240502 08:18:13.585289 19640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.5:41865 every 8 connection(s)
I20240502 08:18:13.587046 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:13.591524 19645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:13.591966 19644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:13.593087 19647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:13.593353 15684 server_base.cc:1034] running on GCE node
I20240502 08:18:13.593554 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:18:13.593602 15684 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:18:13.593638 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714637893593634 us; error 0 us; skew 500 ppm
I20240502 08:18:13.593781 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:13.594434 19641 heartbeater.cc:347] Connected to a master server at 127.15.81.62:39387
I20240502 08:18:13.594570 19641 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:13.595049 19641 heartbeater.cc:511] Master 127.15.81.62:39387 requested a full tablet report, sending...
I20240502 08:18:13.595227 15684 webserver.cc:460] Webserver started at http://127.15.81.6:34289/ using document root <none> and password file <none>
I20240502 08:18:13.595436 15684 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:13.595505 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:13.595666 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:13.595964 19020 ts_manager.cc:194] Registered new tserver with Master: e6313d7fc95e46e88fb241f192368b1e (127.15.81.5:41865)
I20240502 08:18:13.596279 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-5-root/instance:
uuid: "49a329b1dd144199a81b68a046cf8a4d"
format_stamp: "Formatted at 2024-05-02 08:18:13 on dist-test-slave-rhmj"
I20240502 08:18:13.596700 19020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33158
I20240502 08:18:13.599174 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:18:13.600665 19652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:13.600984 15684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 08:18:13.601107 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-5-root
uuid: "49a329b1dd144199a81b68a046cf8a4d"
format_stamp: "Formatted at 2024-05-02 08:18:13 on dist-test-slave-rhmj"
I20240502 08:18:13.601199 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:13.607182 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:18:13.607652 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:13.608187 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:13.608920 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:13.608976 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:13.609040 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:13.609074 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:13.623718 15684 rpc_server.cc:307] RPC server started. Bound to: 127.15.81.6:37361
I20240502 08:18:13.623768 19714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.81.6:37361 every 8 connection(s)
I20240502 08:18:13.630441 19715 heartbeater.cc:347] Connected to a master server at 127.15.81.62:39387
I20240502 08:18:13.630584 19715 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:13.631004 19715 heartbeater.cc:511] Master 127.15.81.62:39387 requested a full tablet report, sending...
I20240502 08:18:13.631808 19020 ts_manager.cc:194] Registered new tserver with Master: 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6:37361)
I20240502 08:18:13.632525 19020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33166
I20240502 08:18:13.894513 19278 consensus_queue.cc:237] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } }
I20240502 08:18:13.896853 19127 raft_consensus.cc:1261] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 08:18:13.897557 19202 raft_consensus.cc:1261] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 08:18:13.897490 19365 consensus_queue.cc:1035] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 08:18:13.898102 19366 consensus_queue.cc:1035] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 08:18:13.900384 19381 raft_consensus.cc:2932] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } } }
I20240502 08:18:13.901484 19202 raft_consensus.cc:2932] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } } }
I20240502 08:18:13.904484 19122 raft_consensus.cc:2932] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } } }
I20240502 08:18:13.907716 19020 catalog_manager.cc:5484] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: config changed from index -1 to 25, NON_VOTER 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) added. New cstate: current_term: 1 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:13.910743 19243 consensus_peers.cc:485] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c -> Peer 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6:37361): Couldn't send request to peer 49a329b1dd144199a81b68a046cf8a4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db356e6346d948938202e72df77e8199. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.914490 19122 consensus_queue.cc:237] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } }
I20240502 08:18:13.917199 19202 raft_consensus.cc:1261] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 08:18:13.917603 19278 raft_consensus.cc:1261] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 08:18:13.918207 19476 consensus_queue.cc:1035] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 08:18:13.918418 19490 consensus_queue.cc:1035] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 08:18:13.920976 19489 raft_consensus.cc:2932] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } } }
I20240502 08:18:13.921602 19202 raft_consensus.cc:2932] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } } }
I20240502 08:18:13.924690 19278 raft_consensus.cc:2932] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } } }
I20240502 08:18:13.928946 19019 catalog_manager.cc:5484] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 reported cstate change: config changed from index -1 to 26, NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) added. New cstate: current_term: 2 leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:13.931340 19122 consensus_queue.cc:237] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } }
I20240502 08:18:13.934346 19202 raft_consensus.cc:1261] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 08:18:13.934895 19332 consensus_queue.cc:1035] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20240502 08:18:13.936120 19093 consensus_peers.cc:485] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 -> Peer 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4:39603): Couldn't send request to peer 4fa05fa6be33496798a23398a84ffbc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fa05fba592744fdbfb354d227626c26. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.936923 19278 raft_consensus.cc:1261] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 08:18:13.937403 19392 consensus_queue.cc:1035] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 08:18:13.937235 19489 raft_consensus.cc:2932] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } } }
I20240502 08:18:13.937811 19198 raft_consensus.cc:2932] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } } }
I20240502 08:18:13.938971 19279 raft_consensus.cc:2932] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } } }
I20240502 08:18:13.944077 19020 catalog_manager.cc:5484] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 reported cstate change: config changed from index -1 to 27, NON_VOTER e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) added. New cstate: current_term: 2 leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } } }
I20240502 08:18:13.944805 19279 consensus_queue.cc:237] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } }
W20240502 08:18:13.947414 19243 consensus_peers.cc:485] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c -> Peer 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6:37361): Couldn't send request to peer 49a329b1dd144199a81b68a046cf8a4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8fa1524bb0a749f9998928d10f4adea6. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.947667 19122 raft_consensus.cc:1261] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 08:18:13.947983 19198 raft_consensus.cc:1261] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 08:18:13.948397 19366 consensus_queue.cc:1035] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 08:18:13.949393 19381 consensus_queue.cc:1035] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
W20240502 08:18:13.950767 19093 consensus_peers.cc:485] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 -> Peer e6313d7fc95e46e88fb241f192368b1e (127.15.81.5:41865): Couldn't send request to peer e6313d7fc95e46e88fb241f192368b1e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4febdd8875b64cc1a156527854a1167a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.952479 19365 raft_consensus.cc:2932] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } } }
I20240502 08:18:13.953209 19198 raft_consensus.cc:2932] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } } }
I20240502 08:18:13.954887 19122 raft_consensus.cc:2932] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } } }
I20240502 08:18:13.960266 19019 catalog_manager.cc:5484] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154 reported cstate change: config changed from index -1 to 25, NON_VOTER 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) added. New cstate: current_term: 1 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } } }
I20240502 08:18:13.967317 19198 consensus_queue.cc:237] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } }
I20240502 08:18:13.969595 19279 raft_consensus.cc:1261] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 08:18:13.969622 19122 raft_consensus.cc:1261] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 08:18:13.970094 19477 consensus_queue.cc:1035] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 08:18:13.970312 19377 consensus_queue.cc:1035] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 08:18:13.972532 19477 raft_consensus.cc:2932] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } } }
I20240502 08:18:13.973479 19122 raft_consensus.cc:2932] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } } }
I20240502 08:18:13.973546 19279 raft_consensus.cc:2932] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } } }
W20240502 08:18:13.977649 19169 consensus_peers.cc:485] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 -> Peer 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4:39603): Couldn't send request to peer 4fa05fa6be33496798a23398a84ffbc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f640265eb3f4b10a0675e3b08b11680. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.979107 19020 catalog_manager.cc:5484] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 reported cstate change: config changed from index -1 to 27, NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) added. New cstate: current_term: 2 leader_uuid: "287e8970c9524c998a437fbb3349fb60" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } } }
I20240502 08:18:13.981544 19122 consensus_queue.cc:237] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } }
W20240502 08:18:13.984091 19093 consensus_peers.cc:485] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 -> Peer e6313d7fc95e46e88fb241f192368b1e (127.15.81.5:41865): Couldn't send request to peer e6313d7fc95e46e88fb241f192368b1e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b48005466a39437b937e2d09c1826d27. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.984189 19198 raft_consensus.cc:1261] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 08:18:13.984421 19279 raft_consensus.cc:1261] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 08:18:13.984691 19489 consensus_queue.cc:1035] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 08:18:13.984938 19332 consensus_queue.cc:1035] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 08:18:13.986968 19332 raft_consensus.cc:2932] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } } }
I20240502 08:18:13.987761 19279 raft_consensus.cc:2932] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } } }
I20240502 08:18:13.992445 19198 raft_consensus.cc:2932] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } } }
I20240502 08:18:13.997555 19018 catalog_manager.cc:5484] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: config changed from index -1 to 27, NON_VOTER e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) added. New cstate: current_term: 2 leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } } }
I20240502 08:18:14.305034 19733 ts_tablet_manager.cc:902] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d: Initiating tablet copy from peer 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:14.306622 19733 tablet_copy_client.cc:308] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d: tablet copy: Beginning tablet copy session from remote peer at address 127.15.81.3:40169
I20240502 08:18:14.312716 19289 tablet_copy_service.cc:140] P 7d576eb536a94138b805dc5ef8b0800c: Received BeginTabletCopySession request for tablet db356e6346d948938202e72df77e8199 from peer 49a329b1dd144199a81b68a046cf8a4d ({username='slave'} at 127.0.0.1:52884)
I20240502 08:18:14.312852 19289 tablet_copy_service.cc:161] P 7d576eb536a94138b805dc5ef8b0800c: Beginning new tablet copy session on tablet db356e6346d948938202e72df77e8199 from peer 49a329b1dd144199a81b68a046cf8a4d at {username='slave'} at 127.0.0.1:52884: session id = 49a329b1dd144199a81b68a046cf8a4d-db356e6346d948938202e72df77e8199
I20240502 08:18:14.315403 19289 tablet_copy_source_session.cc:205] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:14.316530 19733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db356e6346d948938202e72df77e8199. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.321087 19733 tablet_copy_client.cc:783] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:14.321300 19733 tablet_copy_client.cc:647] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:14.322759 19737 tablet_copy_client.cc:1002] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d: tablet copy: Injecting latency into file download: 2000
I20240502 08:18:14.363294 19739 ts_tablet_manager.cc:902] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d: Initiating tablet copy from peer 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:14.364882 19739 tablet_copy_client.cc:308] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d: tablet copy: Beginning tablet copy session from remote peer at address 127.15.81.3:40169
I20240502 08:18:14.365435 19289 tablet_copy_service.cc:140] P 7d576eb536a94138b805dc5ef8b0800c: Received BeginTabletCopySession request for tablet 8fa1524bb0a749f9998928d10f4adea6 from peer 49a329b1dd144199a81b68a046cf8a4d ({username='slave'} at 127.0.0.1:52884)
I20240502 08:18:14.365550 19289 tablet_copy_service.cc:161] P 7d576eb536a94138b805dc5ef8b0800c: Beginning new tablet copy session on tablet 8fa1524bb0a749f9998928d10f4adea6 from peer 49a329b1dd144199a81b68a046cf8a4d at {username='slave'} at 127.0.0.1:52884: session id = 49a329b1dd144199a81b68a046cf8a4d-8fa1524bb0a749f9998928d10f4adea6
I20240502 08:18:14.367693 19289 tablet_copy_source_session.cc:205] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:14.368726 19739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fa1524bb0a749f9998928d10f4adea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.373692 19739 tablet_copy_client.cc:783] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:14.373920 19739 tablet_copy_client.cc:647] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:14.375260 19743 tablet_copy_client.cc:1002] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d: tablet copy: Injecting latency into file download: 2000
I20240502 08:18:14.435339 19744 ts_tablet_manager.cc:902] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e: Initiating tablet copy from peer 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079)
I20240502 08:18:14.436849 19744 tablet_copy_client.cc:308] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e: tablet copy: Beginning tablet copy session from remote peer at address 127.15.81.1:36079
I20240502 08:18:14.443136 19138 tablet_copy_service.cc:140] P 86029a0bbf0b4f77b9c0e96cf121e154: Received BeginTabletCopySession request for tablet b48005466a39437b937e2d09c1826d27 from peer e6313d7fc95e46e88fb241f192368b1e ({username='slave'} at 127.0.0.1:33058)
I20240502 08:18:14.443311 19138 tablet_copy_service.cc:161] P 86029a0bbf0b4f77b9c0e96cf121e154: Beginning new tablet copy session on tablet b48005466a39437b937e2d09c1826d27 from peer e6313d7fc95e46e88fb241f192368b1e at {username='slave'} at 127.0.0.1:33058: session id = e6313d7fc95e46e88fb241f192368b1e-b48005466a39437b937e2d09c1826d27
I20240502 08:18:14.445696 19138 tablet_copy_source_session.cc:205] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:14.446929 19744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b48005466a39437b937e2d09c1826d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.451643 19078 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:18:14.452482 19744 tablet_copy_client.cc:783] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:14.453001 19744 tablet_copy_client.cc:647] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:14.454620 19748 tablet_copy_client.cc:1002] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e: tablet copy: Injecting latency into file download: 2000
I20240502 08:18:14.455091 19278 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db356e6346d948938202e72df77e8199"
dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
mode: GRACEFUL
new_leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
from {username='slave'} at 127.0.0.1:52856
I20240502 08:18:14.455271 19278 raft_consensus.cc:595] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Received request to transfer leadership to TS 86029a0bbf0b4f77b9c0e96cf121e154
I20240502 08:18:14.455711 19078 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 08:18:14.455917 19078 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:18:14.484762 19751 ts_tablet_manager.cc:902] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e: Initiating tablet copy from peer 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079)
I20240502 08:18:14.485971 19751 tablet_copy_client.cc:308] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e: tablet copy: Beginning tablet copy session from remote peer at address 127.15.81.1:36079
I20240502 08:18:14.486528 19138 tablet_copy_service.cc:140] P 86029a0bbf0b4f77b9c0e96cf121e154: Received BeginTabletCopySession request for tablet 4febdd8875b64cc1a156527854a1167a from peer e6313d7fc95e46e88fb241f192368b1e ({username='slave'} at 127.0.0.1:33058)
I20240502 08:18:14.486656 19138 tablet_copy_service.cc:161] P 86029a0bbf0b4f77b9c0e96cf121e154: Beginning new tablet copy session on tablet 4febdd8875b64cc1a156527854a1167a from peer e6313d7fc95e46e88fb241f192368b1e at {username='slave'} at 127.0.0.1:33058: session id = e6313d7fc95e46e88fb241f192368b1e-4febdd8875b64cc1a156527854a1167a
I20240502 08:18:14.489395 19138 tablet_copy_source_session.cc:205] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:14.490602 19751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4febdd8875b64cc1a156527854a1167a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.496731 19751 tablet_copy_client.cc:783] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:14.496955 19751 tablet_copy_client.cc:647] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:14.498283 19755 tablet_copy_client.cc:1002] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e: tablet copy: Injecting latency into file download: 2000
I20240502 08:18:14.524384 19756 ts_tablet_manager.cc:902] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9: Initiating tablet copy from peer 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079)
I20240502 08:18:14.526151 19756 tablet_copy_client.cc:308] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Beginning tablet copy session from remote peer at address 127.15.81.1:36079
I20240502 08:18:14.536887 19138 tablet_copy_service.cc:140] P 86029a0bbf0b4f77b9c0e96cf121e154: Received BeginTabletCopySession request for tablet 5fa05fba592744fdbfb354d227626c26 from peer 4fa05fa6be33496798a23398a84ffbc9 ({username='slave'} at 127.0.0.1:33060)
I20240502 08:18:14.537056 19138 tablet_copy_service.cc:161] P 86029a0bbf0b4f77b9c0e96cf121e154: Beginning new tablet copy session on tablet 5fa05fba592744fdbfb354d227626c26 from peer 4fa05fa6be33496798a23398a84ffbc9 at {username='slave'} at 127.0.0.1:33060: session id = 4fa05fa6be33496798a23398a84ffbc9-5fa05fba592744fdbfb354d227626c26
I20240502 08:18:14.539229 19138 tablet_copy_source_session.cc:205] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:14.540632 19756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa05fba592744fdbfb354d227626c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.545658 19756 tablet_copy_client.cc:783] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:14.545935 19756 tablet_copy_client.cc:647] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:14.547427 19760 tablet_copy_client.cc:1002] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Injecting latency into file download: 2000
I20240502 08:18:14.553939 19566 heartbeater.cc:503] Master 127.15.81.62:39387 was elected leader, sending a full tablet report...
I20240502 08:18:14.594457 19762 ts_tablet_manager.cc:902] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9: Initiating tablet copy from peer 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:14.596825 19762 tablet_copy_client.cc:308] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Beginning tablet copy session from remote peer at address 127.15.81.2:38313
I20240502 08:18:14.598088 19641 heartbeater.cc:503] Master 127.15.81.62:39387 was elected leader, sending a full tablet report...
I20240502 08:18:14.603854 19214 tablet_copy_service.cc:140] P 287e8970c9524c998a437fbb3349fb60: Received BeginTabletCopySession request for tablet 2f640265eb3f4b10a0675e3b08b11680 from peer 4fa05fa6be33496798a23398a84ffbc9 ({username='slave'} at 127.0.0.1:53080)
I20240502 08:18:14.604013 19214 tablet_copy_service.cc:161] P 287e8970c9524c998a437fbb3349fb60: Beginning new tablet copy session on tablet 2f640265eb3f4b10a0675e3b08b11680 from peer 4fa05fa6be33496798a23398a84ffbc9 at {username='slave'} at 127.0.0.1:53080: session id = 4fa05fa6be33496798a23398a84ffbc9-2f640265eb3f4b10a0675e3b08b11680
I20240502 08:18:14.606521 19214 tablet_copy_source_session.cc:205] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:14.607682 19762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f640265eb3f4b10a0675e3b08b11680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.612993 19762 tablet_copy_client.cc:783] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:14.613224 19762 tablet_copy_client.cc:647] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:14.614639 19766 tablet_copy_client.cc:1002] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Injecting latency into file download: 2000
I20240502 08:18:14.633689 19715 heartbeater.cc:503] Master 127.15.81.62:39387 was elected leader, sending a full tablet report...
I20240502 08:18:14.747263 19768 raft_consensus.cc:979] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c: : Instructing follower 86029a0bbf0b4f77b9c0e96cf121e154 to start an election
I20240502 08:18:14.747445 19768 raft_consensus.cc:1067] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Signalling peer 86029a0bbf0b4f77b9c0e96cf121e154 to start an election
I20240502 08:18:14.748008 19127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db356e6346d948938202e72df77e8199"
dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
from {username='slave'} at 127.0.0.1:33030
I20240502 08:18:14.748203 19127 raft_consensus.cc:482] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:14.748302 19127 raft_consensus.cc:3037] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:14.750768 19127 raft_consensus.cc:504] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } }
I20240502 08:18:14.751423 19127 leader_election.cc:288] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [CANDIDATE]: Term 2 election: Requested vote from peers 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:14.751813 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db356e6346d948938202e72df77e8199" candidate_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:14.751864 19278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db356e6346d948938202e72df77e8199" candidate_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
I20240502 08:18:14.752034 19198 raft_consensus.cc:3037] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:14.752046 19278 raft_consensus.cc:3032] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:14.752137 19278 raft_consensus.cc:729] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:14.752406 19278 consensus_queue.cc:260] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } }
I20240502 08:18:14.752712 19278 raft_consensus.cc:3037] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:14.754036 19198 raft_consensus.cc:2454] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86029a0bbf0b4f77b9c0e96cf121e154 in term 2.
I20240502 08:18:14.754447 19092 leader_election.cc:302] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:14.754627 19278 raft_consensus.cc:2454] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86029a0bbf0b4f77b9c0e96cf121e154 in term 2.
I20240502 08:18:14.754745 19489 raft_consensus.cc:2781] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:14.755177 19489 raft_consensus.cc:686] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Becoming Leader. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Running, Role: LEADER
I20240502 08:18:14.755411 19489 consensus_queue.cc:237] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } }
I20240502 08:18:14.757712 19019 catalog_manager.cc:5484] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 reported cstate change: term changed from 1 to 2, leader changed from 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3) to 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1). New cstate: current_term: 2 leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:14.766361 19127 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db356e6346d948938202e72df77e8199"
dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33022
I20240502 08:18:14.766541 19127 raft_consensus.cc:595] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Received request to transfer leadership
I20240502 08:18:14.773171 19127 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db356e6346d948938202e72df77e8199"
dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33022
I20240502 08:18:14.773339 19127 raft_consensus.cc:595] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Received request to transfer leadership
W20240502 08:18:14.773717 19009 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for db356e6346d948938202e72df77e8199 already in progress [suppressed 4 similar messages]
I20240502 08:18:14.774662 19127 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db356e6346d948938202e72df77e8199"
dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33022
I20240502 08:18:14.774793 19127 raft_consensus.cc:595] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Received request to transfer leadership
W20240502 08:18:14.775259 19073 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for db356e6346d948938202e72df77e8199 already in progress
W20240502 08:18:14.775319 19073 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for db356e6346d948938202e72df77e8199 already in progress
I20240502 08:18:15.138314 19278 raft_consensus.cc:1261] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 08:18:15.138826 19769 consensus_queue.cc:1035] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 08:18:15.141096 19774 raft_consensus.cc:979] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154: : Instructing follower 7d576eb536a94138b805dc5ef8b0800c to start an election
I20240502 08:18:15.141225 19774 raft_consensus.cc:1067] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Signalling peer 7d576eb536a94138b805dc5ef8b0800c to start an election
I20240502 08:18:15.141806 19279 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db356e6346d948938202e72df77e8199"
dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
from {username='slave'} at 127.0.0.1:52830
I20240502 08:18:15.142917 19279 raft_consensus.cc:482] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:15.143065 19279 raft_consensus.cc:3037] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:15.146605 19279 raft_consensus.cc:504] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } }
I20240502 08:18:15.149571 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db356e6346d948938202e72df77e8199" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:15.149825 19127 raft_consensus.cc:3032] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:18:15.149919 19127 raft_consensus.cc:729] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Running, Role: LEADER
I20240502 08:18:15.150262 19127 consensus_queue.cc:260] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } }
I20240502 08:18:15.153275 19279 leader_election.cc:288] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 3 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:15.153282 19127 raft_consensus.cc:3037] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:15.153689 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db356e6346d948938202e72df77e8199" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:15.153856 19198 raft_consensus.cc:3037] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:15.156610 19198 raft_consensus.cc:2454] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 3.
I20240502 08:18:15.156980 19243 leader_election.cc:302] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 7d576eb536a94138b805dc5ef8b0800c; no voters:
I20240502 08:18:15.157275 19775 raft_consensus.cc:2781] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:18:15.157400 19775 raft_consensus.cc:686] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 3 LEADER]: Becoming Leader. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:15.157652 19775 consensus_queue.cc:237] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } }
I20240502 08:18:15.158010 19127 raft_consensus.cc:2454] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 3.
I20240502 08:18:15.161118 19019 catalog_manager.cc:5484] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: term changed from 2 to 3, leader changed from 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1) to 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3). New cstate: current_term: 3 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:15.164263 19198 raft_consensus.cc:1226] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Rejecting Update request from peer 86029a0bbf0b4f77b9c0e96cf121e154 for earlier term 2. Current term is 3. Ops: []
I20240502 08:18:15.547050 19198 raft_consensus.cc:1261] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240502 08:18:15.547595 19775 consensus_queue.cc:1035] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 08:18:15.552214 19122 raft_consensus.cc:1261] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 3 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240502 08:18:15.552950 19775 consensus_queue.cc:1035] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20240502 08:18:15.553385 19243 consensus_peers.cc:485] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c -> Peer 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6:37361): Couldn't send request to peer 49a329b1dd144199a81b68a046cf8a4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:16.323135 19733 tablet_copy_client.cc:523] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:16.326305 19733 tablet_bootstrap.cc:492] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d: Bootstrap starting.
I20240502 08:18:16.360314 19733 tablet_bootstrap.cc:492] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=88 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:16.360788 19733 tablet_bootstrap.cc:492] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d: Bootstrap complete.
I20240502 08:18:16.361032 19733 ts_tablet_manager.cc:1367] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.008s
I20240502 08:18:16.361634 19733 raft_consensus.cc:348] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } }
I20240502 08:18:16.361953 19733 raft_consensus.cc:729] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49a329b1dd144199a81b68a046cf8a4d, State: Initialized, Role: LEARNER
I20240502 08:18:16.362210 19733 consensus_queue.cc:260] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } }
I20240502 08:18:16.363026 19733 ts_tablet_manager.cc:1398] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240502 08:18:16.363795 19289 tablet_copy_service.cc:342] P 7d576eb536a94138b805dc5ef8b0800c: Request end of tablet copy session 49a329b1dd144199a81b68a046cf8a4d-db356e6346d948938202e72df77e8199 received from {username='slave'} at 127.0.0.1:52884
I20240502 08:18:16.363907 19289 tablet_copy_service.cc:433] P 7d576eb536a94138b805dc5ef8b0800c: ending tablet copy session 49a329b1dd144199a81b68a046cf8a4d-db356e6346d948938202e72df77e8199 on tablet db356e6346d948938202e72df77e8199 with peer 49a329b1dd144199a81b68a046cf8a4d
I20240502 08:18:16.375598 19739 tablet_copy_client.cc:523] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:16.378615 19739 tablet_bootstrap.cc:492] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d: Bootstrap starting.
I20240502 08:18:16.410305 19739 tablet_bootstrap.cc:492] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=103 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:16.410732 19739 tablet_bootstrap.cc:492] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d: Bootstrap complete.
I20240502 08:18:16.410974 19739 ts_tablet_manager.cc:1367] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.005s
I20240502 08:18:16.411670 19739 raft_consensus.cc:348] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } }
I20240502 08:18:16.411923 19739 raft_consensus.cc:729] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49a329b1dd144199a81b68a046cf8a4d, State: Initialized, Role: LEARNER
I20240502 08:18:16.412122 19739 consensus_queue.cc:260] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } }
I20240502 08:18:16.412678 19739 ts_tablet_manager.cc:1398] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 08:18:16.413282 19289 tablet_copy_service.cc:342] P 7d576eb536a94138b805dc5ef8b0800c: Request end of tablet copy session 49a329b1dd144199a81b68a046cf8a4d-8fa1524bb0a749f9998928d10f4adea6 received from {username='slave'} at 127.0.0.1:52884
I20240502 08:18:16.413370 19289 tablet_copy_service.cc:433] P 7d576eb536a94138b805dc5ef8b0800c: ending tablet copy session 49a329b1dd144199a81b68a046cf8a4d-8fa1524bb0a749f9998928d10f4adea6 on tablet 8fa1524bb0a749f9998928d10f4adea6 with peer 49a329b1dd144199a81b68a046cf8a4d
I20240502 08:18:16.455003 19744 tablet_copy_client.cc:523] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:16.456260 19078 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:18:16.458681 19744 tablet_bootstrap.cc:492] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e: Bootstrap starting.
I20240502 08:18:16.459883 19278 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a21d56d23364d6f802d89fbc035c02f"
dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
mode: GRACEFUL
new_leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
from {username='slave'} at 127.0.0.1:52856
I20240502 08:18:16.460089 19278 raft_consensus.cc:595] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Received request to transfer leadership to TS 86029a0bbf0b4f77b9c0e96cf121e154
I20240502 08:18:16.460502 19078 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 08:18:16.460819 19078 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:18:16.464632 19690 raft_consensus.cc:1203] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25] Dedup: 1.25->[]
I20240502 08:18:16.467862 19690 raft_consensus.cc:3037] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d [term 1 LEARNER]: Advancing to term 3
I20240502 08:18:16.470927 19690 raft_consensus.cc:1261] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d [term 3 LEARNER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 08:18:16.471526 19785 consensus_queue.cc:1037] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 08:18:16.478024 19785 raft_consensus.cc:1050] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c: attempting to promote NON_VOTER 49a329b1dd144199a81b68a046cf8a4d to VOTER
I20240502 08:18:16.478967 19785 consensus_queue.cc:237] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 26, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } }
I20240502 08:18:16.481974 19198 raft_consensus.cc:1261] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 08:18:16.482515 19786 consensus_queue.cc:1035] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 08:18:16.484383 19690 raft_consensus.cc:1261] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d [term 3 LEARNER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 08:18:16.484879 19786 consensus_queue.cc:1035] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 08:18:16.485266 19122 raft_consensus.cc:1261] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 3 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 08:18:16.485796 19365 consensus_queue.cc:1035] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 08:18:16.487290 19365 raft_consensus.cc:2932] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 3 LEADER]: Committing config change with OpId 3.28: config changed from index 25 to 28, 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:16.491425 19690 raft_consensus.cc:2932] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 25 to 28, 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:16.492053 19122 raft_consensus.cc:2932] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 25 to 28, 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:16.497848 19020 catalog_manager.cc:5484] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d reported cstate change: config changed from index 25 to 28, 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:16.497960 19202 raft_consensus.cc:2932] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 25 to 28, 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:16.500620 19751 tablet_copy_client.cc:523] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:16.504901 19744 tablet_bootstrap.cc:492] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=97 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:16.505163 19751 tablet_bootstrap.cc:492] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e: Bootstrap starting.
I20240502 08:18:16.505400 19744 tablet_bootstrap.cc:492] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e: Bootstrap complete.
I20240502 08:18:16.505824 19744 ts_tablet_manager.cc:1367] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e: Time spent bootstrapping tablet: real 0.047s user 0.048s sys 0.000s
I20240502 08:18:16.507239 19744 raft_consensus.cc:348] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } }
I20240502 08:18:16.507445 19744 raft_consensus.cc:729] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e6313d7fc95e46e88fb241f192368b1e, State: Initialized, Role: LEARNER
I20240502 08:18:16.507745 19744 consensus_queue.cc:260] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } }
I20240502 08:18:16.509624 19744 ts_tablet_manager.cc:1398] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:16.510284 19138 tablet_copy_service.cc:342] P 86029a0bbf0b4f77b9c0e96cf121e154: Request end of tablet copy session e6313d7fc95e46e88fb241f192368b1e-b48005466a39437b937e2d09c1826d27 received from {username='slave'} at 127.0.0.1:33058
I20240502 08:18:16.510411 19138 tablet_copy_service.cc:433] P 86029a0bbf0b4f77b9c0e96cf121e154: ending tablet copy session e6313d7fc95e46e88fb241f192368b1e-b48005466a39437b937e2d09c1826d27 on tablet b48005466a39437b937e2d09c1826d27 with peer e6313d7fc95e46e88fb241f192368b1e
I20240502 08:18:16.512292 19278 consensus_queue.cc:237] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } }
I20240502 08:18:16.514454 19690 raft_consensus.cc:1261] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d [term 3 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 08:18:16.514883 19202 raft_consensus.cc:1261] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 08:18:16.515421 19795 consensus_queue.cc:1035] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 08:18:16.515700 19786 consensus_queue.cc:1035] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 08:18:16.518481 19786 raft_consensus.cc:2932] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:16.519577 19202 raft_consensus.cc:2932] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:16.519735 19690 raft_consensus.cc:2932] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:16.523401 19008 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db356e6346d948938202e72df77e8199 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:16.524513 19024 catalog_manager.cc:5484] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 reported cstate change: config changed from index 28 to 29, VOTER 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1) evicted. New cstate: current_term: 3 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:16.530094 19114 tablet_service.cc:1514] Processing DeleteTablet for tablet db356e6346d948938202e72df77e8199 with delete_type TABLET_DATA_TOMBSTONED (TS 86029a0bbf0b4f77b9c0e96cf121e154 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:33022
I20240502 08:18:16.530763 19802 tablet_replica.cc:330] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154: stopping tablet replica
I20240502 08:18:16.530969 19802 raft_consensus.cc:2229] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:16.531308 19802 raft_consensus.cc:2258] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:16.532188 19802 ts_tablet_manager.cc:1875] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:16.538319 19802 ts_tablet_manager.cc:1888] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.28
I20240502 08:18:16.538456 19802 log.cc:1196] T db356e6346d948938202e72df77e8199 P 86029a0bbf0b4f77b9c0e96cf121e154: Deleting WAL directory at /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-0-root/wals/db356e6346d948938202e72df77e8199
I20240502 08:18:16.539269 19009 catalog_manager.cc:4830] TS 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079): tablet db356e6346d948938202e72df77e8199 (table test-workload [id=a1cee04baea8496ba25d89318502a3d4]) successfully deleted
I20240502 08:18:16.547796 19756 tablet_copy_client.cc:523] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:16.550848 19756 tablet_bootstrap.cc:492] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9: Bootstrap starting.
I20240502 08:18:16.557109 19751 tablet_bootstrap.cc:492] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=105 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:16.557567 19751 tablet_bootstrap.cc:492] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e: Bootstrap complete.
I20240502 08:18:16.557868 19751 ts_tablet_manager.cc:1367] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e: Time spent bootstrapping tablet: real 0.053s user 0.043s sys 0.000s
I20240502 08:18:16.558598 19751 raft_consensus.cc:348] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } }
I20240502 08:18:16.558887 19751 raft_consensus.cc:729] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e6313d7fc95e46e88fb241f192368b1e, State: Initialized, Role: LEARNER
I20240502 08:18:16.559180 19751 consensus_queue.cc:260] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } }
I20240502 08:18:16.559845 19751 ts_tablet_manager.cc:1398] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 08:18:16.560499 19138 tablet_copy_service.cc:342] P 86029a0bbf0b4f77b9c0e96cf121e154: Request end of tablet copy session e6313d7fc95e46e88fb241f192368b1e-4febdd8875b64cc1a156527854a1167a received from {username='slave'} at 127.0.0.1:33058
I20240502 08:18:16.560622 19138 tablet_copy_service.cc:433] P 86029a0bbf0b4f77b9c0e96cf121e154: ending tablet copy session e6313d7fc95e46e88fb241f192368b1e-4febdd8875b64cc1a156527854a1167a on tablet 4febdd8875b64cc1a156527854a1167a with peer e6313d7fc95e46e88fb241f192368b1e
I20240502 08:18:16.590472 19756 tablet_bootstrap.cc:492] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=89 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:16.590850 19756 tablet_bootstrap.cc:492] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9: Bootstrap complete.
I20240502 08:18:16.591089 19756 ts_tablet_manager.cc:1367] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9: Time spent bootstrapping tablet: real 0.040s user 0.028s sys 0.009s
I20240502 08:18:16.591737 19756 raft_consensus.cc:348] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } }
I20240502 08:18:16.591950 19756 raft_consensus.cc:729] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4fa05fa6be33496798a23398a84ffbc9, State: Initialized, Role: LEARNER
I20240502 08:18:16.592164 19756 consensus_queue.cc:260] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } }
I20240502 08:18:16.592939 19756 ts_tablet_manager.cc:1398] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 08:18:16.593627 19138 tablet_copy_service.cc:342] P 86029a0bbf0b4f77b9c0e96cf121e154: Request end of tablet copy session 4fa05fa6be33496798a23398a84ffbc9-5fa05fba592744fdbfb354d227626c26 received from {username='slave'} at 127.0.0.1:33060
I20240502 08:18:16.593742 19138 tablet_copy_service.cc:433] P 86029a0bbf0b4f77b9c0e96cf121e154: ending tablet copy session 4fa05fa6be33496798a23398a84ffbc9-5fa05fba592744fdbfb354d227626c26 on tablet 5fa05fba592744fdbfb354d227626c26 with peer 4fa05fa6be33496798a23398a84ffbc9
I20240502 08:18:16.615000 19762 tablet_copy_client.cc:523] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:16.618549 19762 tablet_bootstrap.cc:492] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9: Bootstrap starting.
I20240502 08:18:16.672859 19762 tablet_bootstrap.cc:492] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=103 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:16.673285 19762 tablet_bootstrap.cc:492] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9: Bootstrap complete.
I20240502 08:18:16.673578 19762 ts_tablet_manager.cc:1367] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9: Time spent bootstrapping tablet: real 0.055s user 0.053s sys 0.000s
I20240502 08:18:16.674317 19762 raft_consensus.cc:348] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } }
I20240502 08:18:16.674554 19762 raft_consensus.cc:729] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4fa05fa6be33496798a23398a84ffbc9, State: Initialized, Role: LEARNER
I20240502 08:18:16.674757 19762 consensus_queue.cc:260] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } }
I20240502 08:18:16.675285 19762 ts_tablet_manager.cc:1398] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:16.675999 19214 tablet_copy_service.cc:342] P 287e8970c9524c998a437fbb3349fb60: Request end of tablet copy session 4fa05fa6be33496798a23398a84ffbc9-2f640265eb3f4b10a0675e3b08b11680 received from {username='slave'} at 127.0.0.1:53080
I20240502 08:18:16.676113 19214 tablet_copy_service.cc:433] P 287e8970c9524c998a437fbb3349fb60: ending tablet copy session 4fa05fa6be33496798a23398a84ffbc9-2f640265eb3f4b10a0675e3b08b11680 on tablet 2f640265eb3f4b10a0675e3b08b11680 with peer 4fa05fa6be33496798a23398a84ffbc9
I20240502 08:18:16.724614 19616 raft_consensus.cc:1203] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.26->[2.27-2.27] Dedup: 2.27->[]
I20240502 08:18:16.808207 19541 raft_consensus.cc:1203] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.26->[2.27-2.27] Dedup: 2.27->[]
I20240502 08:18:16.869262 19616 raft_consensus.cc:1203] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.26->[2.27-2.27] Dedup: 2.27->[]
I20240502 08:18:16.876727 19541 raft_consensus.cc:1203] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.25->[2.26-2.26] Dedup: 2.26->[]
I20240502 08:18:17.026963 19786 raft_consensus.cc:979] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c: : Instructing follower 86029a0bbf0b4f77b9c0e96cf121e154 to start an election
I20240502 08:18:17.027221 19786 raft_consensus.cc:1067] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Signalling peer 86029a0bbf0b4f77b9c0e96cf121e154 to start an election
I20240502 08:18:17.027923 19786 raft_consensus.cc:1050] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c: attempting to promote NON_VOTER 49a329b1dd144199a81b68a046cf8a4d to VOTER
I20240502 08:18:17.028981 19786 consensus_queue.cc:237] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } }
I20240502 08:18:17.031209 19122 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a21d56d23364d6f802d89fbc035c02f"
dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
from {username='slave'} at 127.0.0.1:33030
I20240502 08:18:17.031682 19202 raft_consensus.cc:1261] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 08:18:17.031760 19690 raft_consensus.cc:1261] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d [term 1 LEARNER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 08:18:17.032649 19809 consensus_queue.cc:1035] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 08:18:17.033123 19785 consensus_queue.cc:1035] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 08:18:17.035962 19122 raft_consensus.cc:482] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:17.036142 19122 raft_consensus.cc:3037] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:17.036461 19127 raft_consensus.cc:1261] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 08:18:17.037220 19786 consensus_queue.cc:1035] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 08:18:17.040725 19785 raft_consensus.cc:2932] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:17.043939 19690 raft_consensus.cc:2932] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:17.049295 19024 catalog_manager.cc:5484] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: config changed from index 25 to 26, 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:17.049513 19198 raft_consensus.cc:2932] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:17.051069 19122 raft_consensus.cc:504] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:17.053369 19122 leader_election.cc:288] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [CANDIDATE]: Term 2 election: Requested vote from peers 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:17.053875 19202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a21d56d23364d6f802d89fbc035c02f" candidate_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: true dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:17.053344 19126 raft_consensus.cc:2932] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:17.054206 19202 raft_consensus.cc:3037] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:17.054370 19278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a21d56d23364d6f802d89fbc035c02f" candidate_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: true dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
I20240502 08:18:17.054575 19278 raft_consensus.cc:3032] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:17.054651 19278 raft_consensus.cc:729] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:17.054924 19278 consensus_queue.cc:260] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 23, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:17.055176 19278 raft_consensus.cc:3037] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:17.060693 19202 raft_consensus.cc:2454] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86029a0bbf0b4f77b9c0e96cf121e154 in term 2.
I20240502 08:18:17.061206 19092 leader_election.cc:302] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:17.061595 19769 raft_consensus.cc:2781] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:17.062130 19278 raft_consensus.cc:2454] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86029a0bbf0b4f77b9c0e96cf121e154 in term 2.
I20240502 08:18:17.071692 19769 raft_consensus.cc:686] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Becoming Leader. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Running, Role: LEADER
I20240502 08:18:17.072093 19769 consensus_queue.cc:237] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 23, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:17.083981 19018 catalog_manager.cc:5484] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 reported cstate change: term changed from 1 to 2, leader changed from 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3) to 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1). New cstate: current_term: 2 leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:17.085424 19278 consensus_queue.cc:237] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } }
I20240502 08:18:17.087378 19690 raft_consensus.cc:1261] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 08:18:17.087425 19126 raft_consensus.cc:1261] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 08:18:17.087935 19807 consensus_queue.cc:1035] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 08:18:17.088217 19786 consensus_queue.cc:1035] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 08:18:17.091087 19807 raft_consensus.cc:2932] T 8fa1524bb0a749f9998928d10f4adea6 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 287e8970c9524c998a437fbb3349fb60 (127.15.81.2) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:17.091928 19126 raft_consensus.cc:2932] T 8fa1524bb0a749f9998928d10f4adea6 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 287e8970c9524c998a437fbb3349fb60 (127.15.81.2) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:17.093273 19690 raft_consensus.cc:2932] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 287e8970c9524c998a437fbb3349fb60 (127.15.81.2) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:17.098430 19008 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8fa1524bb0a749f9998928d10f4adea6 with cas_config_opid_index 26: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:17.098532 19018 catalog_manager.cc:5484] T 8fa1524bb0a749f9998928d10f4adea6 P 49a329b1dd144199a81b68a046cf8a4d reported cstate change: config changed from index 26 to 27, VOTER 287e8970c9524c998a437fbb3349fb60 (127.15.81.2) evicted. New cstate: current_term: 1 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } }
I20240502 08:18:17.103637 19194 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fa1524bb0a749f9998928d10f4adea6 with delete_type TABLET_DATA_TOMBSTONED (TS 287e8970c9524c998a437fbb3349fb60 not found in new config with opid_index 27) from {username='slave'} at 127.0.0.1:53024
I20240502 08:18:17.104360 19820 tablet_replica.cc:330] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60: stopping tablet replica
I20240502 08:18:17.105669 19820 raft_consensus.cc:2229] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:17.105898 19820 raft_consensus.cc:2258] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:17.107265 19820 ts_tablet_manager.cc:1875] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:17.116359 19820 ts_tablet_manager.cc:1888] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.26
I20240502 08:18:17.116515 19820 log.cc:1196] T 8fa1524bb0a749f9998928d10f4adea6 P 287e8970c9524c998a437fbb3349fb60: Deleting WAL directory at /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-1-root/wals/8fa1524bb0a749f9998928d10f4adea6
I20240502 08:18:17.117389 19010 catalog_manager.cc:4830] TS 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313): tablet 8fa1524bb0a749f9998928d10f4adea6 (table test-workload [id=a1cee04baea8496ba25d89318502a3d4]) successfully deleted
I20240502 08:18:17.172195 19815 raft_consensus.cc:1050] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154: attempting to promote NON_VOTER e6313d7fc95e46e88fb241f192368b1e to VOTER
I20240502 08:18:17.173070 19815 consensus_queue.cc:237] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } }
I20240502 08:18:17.176033 19616 raft_consensus.cc:1261] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [term 2 LEARNER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 08:18:17.176033 19279 raft_consensus.cc:1261] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 08:18:17.176635 19816 consensus_queue.cc:1035] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 08:18:17.177493 19816 consensus_queue.cc:1035] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 08:18:17.188611 19198 raft_consensus.cc:1261] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 08:18:17.190356 19816 consensus_queue.cc:1035] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 08:18:17.195353 19815 raft_consensus.cc:2932] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } }
I20240502 08:18:17.195873 19279 raft_consensus.cc:2932] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } }
I20240502 08:18:17.198863 19198 raft_consensus.cc:2932] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } }
I20240502 08:18:17.201591 19616 raft_consensus.cc:2932] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } }
I20240502 08:18:17.202431 19024 catalog_manager.cc:5484] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: config changed from index 27 to 28, e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } }
I20240502 08:18:17.211313 19126 consensus_queue.cc:237] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } }
I20240502 08:18:17.213553 19616 raft_consensus.cc:1261] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 08:18:17.213553 19198 raft_consensus.cc:1261] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 08:18:17.214416 19769 consensus_queue.cc:1035] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 08:18:17.214727 19816 consensus_queue.cc:1035] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 08:18:17.217325 19816 raft_consensus.cc:2932] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } }
I20240502 08:18:17.217947 19202 raft_consensus.cc:2932] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } }
I20240502 08:18:17.218464 19616 raft_consensus.cc:2932] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } }
I20240502 08:18:17.223273 19009 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b48005466a39437b937e2d09c1826d27 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:17.223490 19019 catalog_manager.cc:5484] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e reported cstate change: config changed from index 28 to 29, VOTER 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3) evicted. New cstate: current_term: 2 leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } }
I20240502 08:18:17.229163 19261 tablet_service.cc:1514] Processing DeleteTablet for tablet b48005466a39437b937e2d09c1826d27 with delete_type TABLET_DATA_TOMBSTONED (TS 7d576eb536a94138b805dc5ef8b0800c not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:52810
I20240502 08:18:17.230126 19826 tablet_replica.cc:330] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c: stopping tablet replica
I20240502 08:18:17.231360 19826 raft_consensus.cc:2229] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:17.231904 19826 raft_consensus.cc:2258] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:17.233304 19826 ts_tablet_manager.cc:1875] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:17.242398 19826 ts_tablet_manager.cc:1888] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.28
I20240502 08:18:17.242686 19826 log.cc:1196] T b48005466a39437b937e2d09c1826d27 P 7d576eb536a94138b805dc5ef8b0800c: Deleting WAL directory at /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-2-root/wals/b48005466a39437b937e2d09c1826d27
I20240502 08:18:17.243619 19008 catalog_manager.cc:4830] TS 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169): tablet b48005466a39437b937e2d09c1826d27 (table test-workload [id=a1cee04baea8496ba25d89318502a3d4]) successfully deleted
I20240502 08:18:17.361531 19828 raft_consensus.cc:1050] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60: attempting to promote NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 to VOTER
I20240502 08:18:17.362269 19828 consensus_queue.cc:237] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } }
I20240502 08:18:17.364881 19279 raft_consensus.cc:1261] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 08:18:17.365628 19541 raft_consensus.cc:1261] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 LEARNER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 08:18:17.365890 19828 consensus_queue.cc:1035] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 08:18:17.366058 19127 raft_consensus.cc:1261] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 08:18:17.366564 19828 consensus_queue.cc:1035] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 08:18:17.366845 19827 consensus_queue.cc:1035] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 08:18:17.385815 19827 raft_consensus.cc:2932] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } }
I20240502 08:18:17.390208 19541 raft_consensus.cc:2932] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } }
I20240502 08:18:17.390434 19126 raft_consensus.cc:2932] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } }
I20240502 08:18:17.390908 19020 catalog_manager.cc:5484] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 reported cstate change: config changed from index 27 to 28, 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "287e8970c9524c998a437fbb3349fb60" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:17.390110 19279 raft_consensus.cc:2932] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } }
I20240502 08:18:17.396345 19769 raft_consensus.cc:1050] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154: attempting to promote NON_VOTER e6313d7fc95e46e88fb241f192368b1e to VOTER
I20240502 08:18:17.397168 19769 consensus_queue.cc:237] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } }
I20240502 08:18:17.402078 19202 consensus_queue.cc:237] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } }
I20240502 08:18:17.404844 19616 raft_consensus.cc:1261] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e [term 2 LEARNER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 08:18:17.406196 19541 raft_consensus.cc:1261] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 08:18:17.406193 19278 raft_consensus.cc:1261] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 08:18:17.406196 19816 consensus_queue.cc:1035] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20240502 08:18:17.407953 19828 consensus_queue.cc:1035] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 08:18:17.409015 19828 consensus_queue.cc:1035] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 08:18:17.410444 19202 raft_consensus.cc:1261] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 08:18:17.411063 19816 consensus_queue.cc:1035] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 08:18:17.412209 19828 raft_consensus.cc:2932] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } }
I20240502 08:18:17.416162 19541 raft_consensus.cc:2932] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } }
I20240502 08:18:17.416563 19278 raft_consensus.cc:2932] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } }
I20240502 08:18:17.422464 19019 catalog_manager.cc:5484] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 reported cstate change: config changed from index 28 to 29, VOTER 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1) evicted. New cstate: current_term: 2 leader_uuid: "287e8970c9524c998a437fbb3349fb60" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } }
I20240502 08:18:17.423748 19279 raft_consensus.cc:1261] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 08:18:17.425107 19816 consensus_queue.cc:1035] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 08:18:17.426124 19816 raft_consensus.cc:2932] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } }
I20240502 08:18:17.427531 19114 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f640265eb3f4b10a0675e3b08b11680 with delete_type TABLET_DATA_TOMBSTONED (TS 86029a0bbf0b4f77b9c0e96cf121e154 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:33022
I20240502 08:18:17.428560 19010 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2f640265eb3f4b10a0675e3b08b11680 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:17.428790 19615 raft_consensus.cc:2932] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } }
I20240502 08:18:17.434767 19019 catalog_manager.cc:5484] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e reported cstate change: config changed from index 27 to 28, e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } }
I20240502 08:18:17.436134 19202 raft_consensus.cc:2932] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } }
I20240502 08:18:17.442682 19126 consensus_queue.cc:237] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } }
I20240502 08:18:17.444249 19837 tablet_replica.cc:330] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154: stopping tablet replica
I20240502 08:18:17.444566 19837 raft_consensus.cc:2229] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:17.444957 19837 raft_consensus.cc:2258] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:17.446671 19616 raft_consensus.cc:1261] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 08:18:17.447377 19837 ts_tablet_manager.cc:1875] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:17.447533 19816 consensus_queue.cc:1035] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 08:18:17.449707 19198 raft_consensus.cc:1261] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 08:18:17.450583 19816 consensus_queue.cc:1035] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 08:18:17.453285 19822 raft_consensus.cc:2932] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } }
I20240502 08:18:17.456984 19615 raft_consensus.cc:2932] T 4febdd8875b64cc1a156527854a1167a P e6313d7fc95e46e88fb241f192368b1e [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } }
I20240502 08:18:17.460264 19020 catalog_manager.cc:5484] T 4febdd8875b64cc1a156527854a1167a P 86029a0bbf0b4f77b9c0e96cf121e154 reported cstate change: config changed from index 28 to 29, VOTER 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3) evicted. New cstate: current_term: 2 leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:17.461145 19009 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4febdd8875b64cc1a156527854a1167a with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:17.461952 19202 raft_consensus.cc:2932] T 4febdd8875b64cc1a156527854a1167a P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } }
I20240502 08:18:17.467197 19261 tablet_service.cc:1514] Processing DeleteTablet for tablet 4febdd8875b64cc1a156527854a1167a with delete_type TABLET_DATA_TOMBSTONED (TS 7d576eb536a94138b805dc5ef8b0800c not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:52810
I20240502 08:18:17.467542 19826 tablet_replica.cc:330] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c: stopping tablet replica
I20240502 08:18:17.467849 19826 raft_consensus.cc:2229] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:17.468117 19826 pending_rounds.cc:62] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c: Trying to abort 1 pending ops.
I20240502 08:18:17.468263 19826 pending_rounds.cc:68] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c: Aborting op as it isn't in flight: id { term: 2 index: 28 } timestamp: 7023156827736702976 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "4febdd8875b64cc1a156527854a1167a" old_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } } } new_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } } } }
I20240502 08:18:17.468825 19826 raft_consensus.cc:2894] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Aborting config change with OpId 2.28: Aborted: Op aborted
I20240502 08:18:17.468943 19826 raft_consensus.cc:2866] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 08:18:17.469043 19826 raft_consensus.cc:2258] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:17.472380 19826 ts_tablet_manager.cc:1875] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:17.481241 19769 raft_consensus.cc:1050] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154: attempting to promote NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 to VOTER
I20240502 08:18:17.481294 19826 ts_tablet_manager.cc:1888] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.28
I20240502 08:18:17.481832 19826 log.cc:1196] T 4febdd8875b64cc1a156527854a1167a P 7d576eb536a94138b805dc5ef8b0800c: Deleting WAL directory at /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-2-root/wals/4febdd8875b64cc1a156527854a1167a
I20240502 08:18:17.481945 19769 consensus_queue.cc:237] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } }
I20240502 08:18:17.481695 19837 ts_tablet_manager.cc:1888] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.28
I20240502 08:18:17.482692 19837 log.cc:1196] T 2f640265eb3f4b10a0675e3b08b11680 P 86029a0bbf0b4f77b9c0e96cf121e154: Deleting WAL directory at /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-0-root/wals/2f640265eb3f4b10a0675e3b08b11680
I20240502 08:18:17.483211 19008 catalog_manager.cc:4830] TS 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169): tablet 4febdd8875b64cc1a156527854a1167a (table test-workload [id=a1cee04baea8496ba25d89318502a3d4]) successfully deleted
I20240502 08:18:17.483521 19009 catalog_manager.cc:4830] TS 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079): tablet 2f640265eb3f4b10a0675e3b08b11680 (table test-workload [id=a1cee04baea8496ba25d89318502a3d4]) successfully deleted
I20240502 08:18:17.484526 19541 raft_consensus.cc:1261] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 LEARNER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 08:18:17.485046 19822 consensus_queue.cc:1035] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 08:18:17.486009 19278 raft_consensus.cc:1261] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 08:18:17.486176 19202 raft_consensus.cc:1261] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 08:18:17.486799 19822 consensus_queue.cc:1035] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 08:18:17.487056 19816 consensus_queue.cc:1035] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 08:18:17.493670 19822 raft_consensus.cc:2932] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index 26 to 27, 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } }
I20240502 08:18:17.496218 19278 raft_consensus.cc:2932] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } }
I20240502 08:18:17.499543 19019 catalog_manager.cc:5484] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 reported cstate change: config changed from index 26 to 27, 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:17.499931 19198 raft_consensus.cc:2932] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } }
I20240502 08:18:17.508515 19541 raft_consensus.cc:2932] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } }
I20240502 08:18:17.512645 19126 consensus_queue.cc:237] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } }
I20240502 08:18:17.515870 19278 raft_consensus.cc:1261] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 08:18:17.516448 19816 consensus_queue.cc:1035] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 08:18:17.517109 19540 raft_consensus.cc:1261] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 08:18:17.517911 19816 consensus_queue.cc:1035] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 08:18:17.519393 19822 raft_consensus.cc:2932] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER 287e8970c9524c998a437fbb3349fb60 (127.15.81.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } }
I20240502 08:18:17.521713 19540 raft_consensus.cc:2932] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER 287e8970c9524c998a437fbb3349fb60 (127.15.81.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } }
I20240502 08:18:17.524050 19009 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5fa05fba592744fdbfb354d227626c26 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:17.525130 19278 raft_consensus.cc:2932] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER 287e8970c9524c998a437fbb3349fb60 (127.15.81.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } }
I20240502 08:18:17.525622 19020 catalog_manager.cc:5484] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 reported cstate change: config changed from index 27 to 28, VOTER 287e8970c9524c998a437fbb3349fb60 (127.15.81.2) evicted. New cstate: current_term: 2 leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:17.530261 19194 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fa05fba592744fdbfb354d227626c26 with delete_type TABLET_DATA_TOMBSTONED (TS 287e8970c9524c998a437fbb3349fb60 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:53024
I20240502 08:18:17.530762 19820 tablet_replica.cc:330] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60: stopping tablet replica
I20240502 08:18:17.530956 19820 raft_consensus.cc:2229] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:17.531268 19820 raft_consensus.cc:2258] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:17.532433 19820 ts_tablet_manager.cc:1875] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:17.536890 19820 ts_tablet_manager.cc:1888] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.27
I20240502 08:18:17.537073 19820 log.cc:1196] T 5fa05fba592744fdbfb354d227626c26 P 287e8970c9524c998a437fbb3349fb60: Deleting WAL directory at /tmp/dist-test-taskghyKOZ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714637795017386-15684-0/minicluster-data/ts-1-root/wals/5fa05fba592744fdbfb354d227626c26
I20240502 08:18:17.537806 19010 catalog_manager.cc:4830] TS 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313): tablet 5fa05fba592744fdbfb354d227626c26 (table test-workload [id=a1cee04baea8496ba25d89318502a3d4]) successfully deleted
I20240502 08:18:17.544616 19198 raft_consensus.cc:1261] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20240502 08:18:17.545194 19769 consensus_queue.cc:1035] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240502 08:18:17.556613 19278 raft_consensus.cc:1261] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20240502 08:18:17.557442 19769 consensus_queue.cc:1035] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240502 08:18:18.461207 19078 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:18:18.470216 19202 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f640265eb3f4b10a0675e3b08b11680"
dest_uuid: "287e8970c9524c998a437fbb3349fb60"
mode: GRACEFUL
new_leader_uuid: "4fa05fa6be33496798a23398a84ffbc9"
from {username='slave'} at 127.0.0.1:53082
I20240502 08:18:18.470443 19202 raft_consensus.cc:595] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 2 LEADER]: Received request to transfer leadership to TS 4fa05fa6be33496798a23398a84ffbc9
I20240502 08:18:18.471535 19278 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "47ec01cade6e41d1a0ff6c8929ee8155"
dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
mode: GRACEFUL
new_leader_uuid: "287e8970c9524c998a437fbb3349fb60"
from {username='slave'} at 127.0.0.1:52856
I20240502 08:18:18.471747 19278 raft_consensus.cc:595] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Received request to transfer leadership to TS 287e8970c9524c998a437fbb3349fb60
I20240502 08:18:18.477443 19126 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5fa05fba592744fdbfb354d227626c26"
dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
mode: GRACEFUL
new_leader_uuid: "7d576eb536a94138b805dc5ef8b0800c"
from {username='slave'} at 127.0.0.1:33070
I20240502 08:18:18.477645 19126 raft_consensus.cc:595] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Received request to transfer leadership to TS 7d576eb536a94138b805dc5ef8b0800c
I20240502 08:18:18.478595 19126 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a21d56d23364d6f802d89fbc035c02f"
dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
mode: GRACEFUL
new_leader_uuid: "287e8970c9524c998a437fbb3349fb60"
from {username='slave'} at 127.0.0.1:33070
I20240502 08:18:18.478730 19126 raft_consensus.cc:595] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Received request to transfer leadership to TS 287e8970c9524c998a437fbb3349fb60
I20240502 08:18:18.479532 19126 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b48005466a39437b937e2d09c1826d27"
dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
mode: GRACEFUL
new_leader_uuid: "e6313d7fc95e46e88fb241f192368b1e"
from {username='slave'} at 127.0.0.1:33070
I20240502 08:18:18.479720 19126 raft_consensus.cc:595] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Received request to transfer leadership to TS e6313d7fc95e46e88fb241f192368b1e
I20240502 08:18:18.480643 19202 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e844ad43672042da8eabbc9b74e622f7"
dest_uuid: "287e8970c9524c998a437fbb3349fb60"
mode: GRACEFUL
new_leader_uuid: "7d576eb536a94138b805dc5ef8b0800c"
from {username='slave'} at 127.0.0.1:53082
I20240502 08:18:18.480788 19202 raft_consensus.cc:595] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 1 LEADER]: Received request to transfer leadership to TS 7d576eb536a94138b805dc5ef8b0800c
I20240502 08:18:18.481271 19078 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240502 08:18:18.481520 19078 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:18:18.500022 19854 raft_consensus.cc:979] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60: : Instructing follower 4fa05fa6be33496798a23398a84ffbc9 to start an election
I20240502 08:18:18.500174 19854 raft_consensus.cc:1067] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 2 LEADER]: Signalling peer 4fa05fa6be33496798a23398a84ffbc9 to start an election
I20240502 08:18:18.500797 19540 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2f640265eb3f4b10a0675e3b08b11680"
dest_uuid: "4fa05fa6be33496798a23398a84ffbc9"
from {username='slave'} at 127.0.0.1:52068
I20240502 08:18:18.500973 19540 raft_consensus.cc:482] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:18.501061 19540 raft_consensus.cc:3037] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:18.503044 19540 raft_consensus.cc:504] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } }
I20240502 08:18:18.503777 19540 leader_election.cc:288] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [CANDIDATE]: Term 3 election: Requested vote from peers 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:18.504206 19202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f640265eb3f4b10a0675e3b08b11680" candidate_uuid: "4fa05fa6be33496798a23398a84ffbc9" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:18.504403 19202 raft_consensus.cc:3032] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:18:18.504487 19202 raft_consensus.cc:729] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Running, Role: LEADER
I20240502 08:18:18.504805 19202 consensus_queue.cc:260] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } }
I20240502 08:18:18.505105 19202 raft_consensus.cc:3037] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:18.507061 19202 raft_consensus.cc:2454] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fa05fa6be33496798a23398a84ffbc9 in term 3.
I20240502 08:18:18.507440 19505 leader_election.cc:302] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 4fa05fa6be33496798a23398a84ffbc9; no voters:
I20240502 08:18:18.508034 19857 raft_consensus.cc:2781] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:18:18.508405 19857 raft_consensus.cc:686] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [term 3 LEADER]: Becoming Leader. State: Replica: 4fa05fa6be33496798a23398a84ffbc9, State: Running, Role: LEADER
I20240502 08:18:18.508773 19857 consensus_queue.cc:237] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } }
I20240502 08:18:18.509977 19278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f640265eb3f4b10a0675e3b08b11680" candidate_uuid: "4fa05fa6be33496798a23398a84ffbc9" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
I20240502 08:18:18.510169 19278 raft_consensus.cc:3037] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:18.511245 19019 catalog_manager.cc:5484] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 reported cstate change: term changed from 2 to 3, leader changed from 287e8970c9524c998a437fbb3349fb60 (127.15.81.2) to 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4). New cstate: current_term: 3 leader_uuid: "4fa05fa6be33496798a23398a84ffbc9" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:18.512509 19278 raft_consensus.cc:2454] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fa05fa6be33496798a23398a84ffbc9 in term 3.
W20240502 08:18:18.530336 19073 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
W20240502 08:18:18.531778 19073 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet e844ad43672042da8eabbc9b74e622f7, TS 86029a0bbf0b4f77b9c0e96cf121e154 -> TS 49a329b1dd144199a81b68a046cf8a4d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
W20240502 08:18:18.531862 19073 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet e844ad43672042da8eabbc9b74e622f7, TS 86029a0bbf0b4f77b9c0e96cf121e154 -> TS 49a329b1dd144199a81b68a046cf8a4d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
W20240502 08:18:18.533097 19073 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 5990222a54794361819bebe9e51582e5, TS 287e8970c9524c998a437fbb3349fb60 -> TS e6313d7fc95e46e88fb241f192368b1e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
W20240502 08:18:18.533165 19073 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 5990222a54794361819bebe9e51582e5, TS 287e8970c9524c998a437fbb3349fb60 -> TS e6313d7fc95e46e88fb241f192368b1e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
W20240502 08:18:18.534248 19073 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 7d7253eb5fb84187805cff227d030f54, TS 7d576eb536a94138b805dc5ef8b0800c -> TS 4fa05fa6be33496798a23398a84ffbc9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
W20240502 08:18:18.534322 19073 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 7d7253eb5fb84187805cff227d030f54, TS 7d576eb536a94138b805dc5ef8b0800c -> TS 4fa05fa6be33496798a23398a84ffbc9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
W20240502 08:18:18.535746 19073 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 47ec01cade6e41d1a0ff6c8929ee8155, TS 86029a0bbf0b4f77b9c0e96cf121e154 -> TS 49a329b1dd144199a81b68a046cf8a4d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
W20240502 08:18:18.535845 19073 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 47ec01cade6e41d1a0ff6c8929ee8155, TS 86029a0bbf0b4f77b9c0e96cf121e154 -> TS 49a329b1dd144199a81b68a046cf8a4d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
W20240502 08:18:18.537046 19073 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet db356e6346d948938202e72df77e8199, TS 287e8970c9524c998a437fbb3349fb60 -> TS 4fa05fa6be33496798a23398a84ffbc9 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } }
W20240502 08:18:18.537107 19073 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet db356e6346d948938202e72df77e8199, TS 287e8970c9524c998a437fbb3349fb60 -> TS 4fa05fa6be33496798a23398a84ffbc9 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } }
W20240502 08:18:18.538264 19073 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d3591d08cc604b349553a185225ef943, TS 7d576eb536a94138b805dc5ef8b0800c -> TS e6313d7fc95e46e88fb241f192368b1e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
W20240502 08:18:18.538331 19073 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d3591d08cc604b349553a185225ef943, TS 7d576eb536a94138b805dc5ef8b0800c -> TS e6313d7fc95e46e88fb241f192368b1e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:18.552313 19850 raft_consensus.cc:979] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154: : Instructing follower 7d576eb536a94138b805dc5ef8b0800c to start an election
I20240502 08:18:18.552516 19850 raft_consensus.cc:1067] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Signalling peer 7d576eb536a94138b805dc5ef8b0800c to start an election
I20240502 08:18:18.553114 19278 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5fa05fba592744fdbfb354d227626c26"
dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
from {username='slave'} at 127.0.0.1:52830
I20240502 08:18:18.553261 19278 raft_consensus.cc:482] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:18.553328 19278 raft_consensus.cc:3037] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:18.555197 19278 raft_consensus.cc:504] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } }
I20240502 08:18:18.555909 19278 leader_election.cc:288] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 3 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4:39603)
I20240502 08:18:18.556576 19126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa05fba592744fdbfb354d227626c26" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:18.556782 19126 raft_consensus.cc:3032] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:18:18.556864 19126 raft_consensus.cc:729] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Running, Role: LEADER
I20240502 08:18:18.557096 19126 consensus_queue.cc:260] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } }
I20240502 08:18:18.557432 19126 raft_consensus.cc:3037] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:18.559931 19126 raft_consensus.cc:2454] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 3.
I20240502 08:18:18.560464 19242 leader_election.cc:302] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:18.560748 19785 raft_consensus.cc:2781] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:18:18.561368 19785 raft_consensus.cc:686] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [term 3 LEADER]: Becoming Leader. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:18.561635 19785 consensus_queue.cc:237] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } }
I20240502 08:18:18.563323 19540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa05fba592744fdbfb354d227626c26" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "4fa05fa6be33496798a23398a84ffbc9"
I20240502 08:18:18.563517 19540 raft_consensus.cc:3037] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:18.564411 19019 catalog_manager.cc:5484] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: term changed from 2 to 3, leader changed from 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1) to 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3). New cstate: current_term: 3 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:18.566033 19540 raft_consensus.cc:2454] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 3.
I20240502 08:18:18.606530 19865 raft_consensus.cc:979] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c: : Instructing follower 287e8970c9524c998a437fbb3349fb60 to start an election
I20240502 08:18:18.606688 19865 raft_consensus.cc:1067] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Signalling peer 287e8970c9524c998a437fbb3349fb60 to start an election
I20240502 08:18:18.607255 19202 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47ec01cade6e41d1a0ff6c8929ee8155"
dest_uuid: "287e8970c9524c998a437fbb3349fb60"
from {username='slave'} at 127.0.0.1:53040
I20240502 08:18:18.607419 19202 raft_consensus.cc:482] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:18.607533 19202 raft_consensus.cc:3037] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:18.609491 19202 raft_consensus.cc:504] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:18.610037 19202 leader_election.cc:288] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 2 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:18.610457 19126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ec01cade6e41d1a0ff6c8929ee8155" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:18.610452 19278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ec01cade6e41d1a0ff6c8929ee8155" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
I20240502 08:18:18.610702 19278 raft_consensus.cc:3032] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:18.610702 19126 raft_consensus.cc:3037] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:18.610810 19278 raft_consensus.cc:729] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:18.611054 19278 consensus_queue.cc:260] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:18.611438 19278 raft_consensus.cc:3037] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:18.613543 19126 raft_consensus.cc:2454] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 2.
I20240502 08:18:18.613958 19278 raft_consensus.cc:2454] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 2.
I20240502 08:18:18.613937 19167 leader_election.cc:302] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:18.614233 19854 raft_consensus.cc:2781] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:18.614364 19854 raft_consensus.cc:686] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [term 2 LEADER]: Becoming Leader. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Running, Role: LEADER
I20240502 08:18:18.614650 19854 consensus_queue.cc:237] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:18.617163 19019 catalog_manager.cc:5484] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 reported cstate change: term changed from 1 to 2, leader changed from 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3) to 287e8970c9524c998a437fbb3349fb60 (127.15.81.2). New cstate: current_term: 2 leader_uuid: "287e8970c9524c998a437fbb3349fb60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:18.667316 19769 raft_consensus.cc:979] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154: : Instructing follower 287e8970c9524c998a437fbb3349fb60 to start an election
I20240502 08:18:18.667486 19769 raft_consensus.cc:1067] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Signalling peer 287e8970c9524c998a437fbb3349fb60 to start an election
I20240502 08:18:18.668174 19202 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a21d56d23364d6f802d89fbc035c02f"
dest_uuid: "287e8970c9524c998a437fbb3349fb60"
from {username='slave'} at 127.0.0.1:53056
I20240502 08:18:18.668375 19202 raft_consensus.cc:482] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:18.668479 19202 raft_consensus.cc:3037] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:18.670444 19202 raft_consensus.cc:504] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:18.671066 19202 leader_election.cc:288] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 3 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
I20240502 08:18:18.671464 19126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a21d56d23364d6f802d89fbc035c02f" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 3 candidate_status { last_received { term: 2 index: 24 } } ignore_live_leader: true dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:18.671554 19278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a21d56d23364d6f802d89fbc035c02f" candidate_uuid: "287e8970c9524c998a437fbb3349fb60" candidate_term: 3 candidate_status { last_received { term: 2 index: 24 } } ignore_live_leader: true dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
I20240502 08:18:18.671684 19126 raft_consensus.cc:3032] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:18:18.671695 19278 raft_consensus.cc:3037] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:18.671777 19126 raft_consensus.cc:729] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Running, Role: LEADER
I20240502 08:18:18.672041 19126 consensus_queue.cc:260] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 2.24, Last appended by leader: 24, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:18.672367 19126 raft_consensus.cc:3037] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:18.674382 19278 raft_consensus.cc:2454] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 3.
I20240502 08:18:18.674758 19166 leader_election.cc:302] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287e8970c9524c998a437fbb3349fb60, 7d576eb536a94138b805dc5ef8b0800c; no voters:
I20240502 08:18:18.674770 19126 raft_consensus.cc:2454] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287e8970c9524c998a437fbb3349fb60 in term 3.
I20240502 08:18:18.675021 19854 raft_consensus.cc:2781] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:18:18.675145 19854 raft_consensus.cc:686] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [term 3 LEADER]: Becoming Leader. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Running, Role: LEADER
I20240502 08:18:18.675366 19854 consensus_queue.cc:237] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 2.24, Last appended by leader: 24, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:18.677842 19019 catalog_manager.cc:5484] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 reported cstate change: term changed from 2 to 3, leader changed from 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1) to 287e8970c9524c998a437fbb3349fb60 (127.15.81.2). New cstate: current_term: 3 leader_uuid: "287e8970c9524c998a437fbb3349fb60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:18.814111 19850 raft_consensus.cc:979] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154: : Instructing follower e6313d7fc95e46e88fb241f192368b1e to start an election
I20240502 08:18:18.814257 19850 raft_consensus.cc:1067] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Signalling peer e6313d7fc95e46e88fb241f192368b1e to start an election
I20240502 08:18:18.814939 19615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b48005466a39437b937e2d09c1826d27"
dest_uuid: "e6313d7fc95e46e88fb241f192368b1e"
from {username='slave'} at 127.0.0.1:38422
I20240502 08:18:18.815122 19615 raft_consensus.cc:482] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:18.815204 19615 raft_consensus.cc:3037] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:18.817152 19615 raft_consensus.cc:504] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } }
I20240502 08:18:18.817834 19615 leader_election.cc:288] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [CANDIDATE]: Term 3 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:18.818236 19126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48005466a39437b937e2d09c1826d27" candidate_uuid: "e6313d7fc95e46e88fb241f192368b1e" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:18.818681 19126 raft_consensus.cc:3032] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:18:18.818768 19126 raft_consensus.cc:729] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 86029a0bbf0b4f77b9c0e96cf121e154, State: Running, Role: LEADER
I20240502 08:18:18.819046 19126 consensus_queue.cc:260] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } }
I20240502 08:18:18.819413 19126 raft_consensus.cc:3037] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:18.821681 19126 raft_consensus.cc:2454] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6313d7fc95e46e88fb241f192368b1e in term 3.
I20240502 08:18:18.822081 19580 leader_election.cc:302] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86029a0bbf0b4f77b9c0e96cf121e154, e6313d7fc95e46e88fb241f192368b1e; no voters:
I20240502 08:18:18.822726 19870 raft_consensus.cc:2781] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:18:18.823146 19870 raft_consensus.cc:686] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [term 3 LEADER]: Becoming Leader. State: Replica: e6313d7fc95e46e88fb241f192368b1e, State: Running, Role: LEADER
I20240502 08:18:18.823423 19870 consensus_queue.cc:237] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } }
I20240502 08:18:18.823808 19202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48005466a39437b937e2d09c1826d27" candidate_uuid: "e6313d7fc95e46e88fb241f192368b1e" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:18.823999 19202 raft_consensus.cc:3037] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:18.826227 19202 raft_consensus.cc:2454] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6313d7fc95e46e88fb241f192368b1e in term 3.
I20240502 08:18:18.826480 19019 catalog_manager.cc:5484] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e reported cstate change: term changed from 2 to 3, leader changed from 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1) to e6313d7fc95e46e88fb241f192368b1e (127.15.81.5). New cstate: current_term: 3 leader_uuid: "e6313d7fc95e46e88fb241f192368b1e" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:18.958065 19540 raft_consensus.cc:1261] T 5fa05fba592744fdbfb354d227626c26 P 4fa05fa6be33496798a23398a84ffbc9 [term 3 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 08:18:18.958616 19865 consensus_queue.cc:1035] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 08:18:18.961664 19202 raft_consensus.cc:1261] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Refusing update from remote peer 4fa05fa6be33496798a23398a84ffbc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 08:18:18.962422 19857 consensus_queue.cc:1035] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 08:18:18.970839 19278 raft_consensus.cc:1261] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 3 FOLLOWER]: Refusing update from remote peer 4fa05fa6be33496798a23398a84ffbc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 08:18:18.971711 19858 consensus_queue.cc:1035] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 08:18:18.974647 19126 raft_consensus.cc:1261] T 5fa05fba592744fdbfb354d227626c26 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 3 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 08:18:18.975107 19785 consensus_queue.cc:1035] T 5fa05fba592744fdbfb354d227626c26 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 08:18:19.043666 19828 raft_consensus.cc:979] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60: : Instructing follower 7d576eb536a94138b805dc5ef8b0800c to start an election
I20240502 08:18:19.043884 19828 raft_consensus.cc:1067] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 1 LEADER]: Signalling peer 7d576eb536a94138b805dc5ef8b0800c to start an election
I20240502 08:18:19.044539 19278 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e844ad43672042da8eabbc9b74e622f7"
dest_uuid: "7d576eb536a94138b805dc5ef8b0800c"
from {username='slave'} at 127.0.0.1:52814
I20240502 08:18:19.044775 19278 raft_consensus.cc:482] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:19.044880 19278 raft_consensus.cc:3037] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:19.046911 19278 raft_consensus.cc:504] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:19.047478 19278 leader_election.cc:288] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 2 election: Requested vote from peers 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079), 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:19.047849 19126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e844ad43672042da8eabbc9b74e622f7" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "86029a0bbf0b4f77b9c0e96cf121e154"
I20240502 08:18:19.047930 19202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e844ad43672042da8eabbc9b74e622f7" candidate_uuid: "7d576eb536a94138b805dc5ef8b0800c" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "287e8970c9524c998a437fbb3349fb60"
I20240502 08:18:19.048045 19126 raft_consensus.cc:3037] T e844ad43672042da8eabbc9b74e622f7 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:19.048110 19202 raft_consensus.cc:3032] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:19.048252 19202 raft_consensus.cc:729] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 287e8970c9524c998a437fbb3349fb60, State: Running, Role: LEADER
I20240502 08:18:19.048480 19202 consensus_queue.cc:260] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:19.048768 19202 raft_consensus.cc:3037] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:19.049934 19126 raft_consensus.cc:2454] T e844ad43672042da8eabbc9b74e622f7 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 2.
I20240502 08:18:19.050273 19242 leader_election.cc:302] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d576eb536a94138b805dc5ef8b0800c, 86029a0bbf0b4f77b9c0e96cf121e154; no voters:
I20240502 08:18:19.050510 19785 raft_consensus.cc:2781] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:19.050623 19785 raft_consensus.cc:686] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [term 2 LEADER]: Becoming Leader. State: Replica: 7d576eb536a94138b805dc5ef8b0800c, State: Running, Role: LEADER
I20240502 08:18:19.050671 19202 raft_consensus.cc:2454] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d576eb536a94138b805dc5ef8b0800c in term 2.
I20240502 08:18:19.050864 19785 consensus_queue.cc:237] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } }
I20240502 08:18:19.053419 19019 catalog_manager.cc:5484] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: term changed from 1 to 2, leader changed from 287e8970c9524c998a437fbb3349fb60 (127.15.81.2) to 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3). New cstate: current_term: 2 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:19.099505 19278 raft_consensus.cc:1261] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 08:18:19.100155 19828 consensus_queue.cc:1035] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 08:18:19.103746 19126 raft_consensus.cc:1261] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 08:18:19.104415 19828 consensus_queue.cc:1035] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 08:18:19.141628 19278 raft_consensus.cc:1261] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 3 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 2 index: 24. Preceding OpId from leader: term: 3 index: 25. (index mismatch)
I20240502 08:18:19.142184 19828 consensus_queue.cc:1035] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 08:18:19.145539 19126 raft_consensus.cc:1261] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 3 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 2 index: 24. Preceding OpId from leader: term: 3 index: 25. (index mismatch)
I20240502 08:18:19.146314 19854 consensus_queue.cc:1035] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 08:18:19.373198 19202 raft_consensus.cc:1261] T b48005466a39437b937e2d09c1826d27 P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Refusing update from remote peer e6313d7fc95e46e88fb241f192368b1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 08:18:19.373679 19886 consensus_queue.cc:1035] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 08:18:19.378113 19126 raft_consensus.cc:1261] T b48005466a39437b937e2d09c1826d27 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 3 FOLLOWER]: Refusing update from remote peer e6313d7fc95e46e88fb241f192368b1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 08:18:19.378717 19888 consensus_queue.cc:1035] T b48005466a39437b937e2d09c1826d27 P e6313d7fc95e46e88fb241f192368b1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 08:18:19.484660 19202 raft_consensus.cc:1261] T e844ad43672042da8eabbc9b74e622f7 P 287e8970c9524c998a437fbb3349fb60 [term 2 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 08:18:19.485255 19892 consensus_queue.cc:1035] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 08:18:19.488615 19126 raft_consensus.cc:1261] T e844ad43672042da8eabbc9b74e622f7 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 08:18:19.489455 19865 consensus_queue.cc:1035] T e844ad43672042da8eabbc9b74e622f7 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 08:18:19.544438 19202 consensus_queue.cc:237] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } }
I20240502 08:18:19.546401 19126 raft_consensus.cc:1261] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 08:18:19.546635 19278 raft_consensus.cc:1261] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
W20240502 08:18:19.546674 19169 consensus_peers.cc:485] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 -> Peer 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4:39603): Couldn't send request to peer 4fa05fa6be33496798a23398a84ffbc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47ec01cade6e41d1a0ff6c8929ee8155. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:19.547000 19828 consensus_queue.cc:1035] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 08:18:19.547262 19896 consensus_queue.cc:1035] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 08:18:19.549525 19854 raft_consensus.cc:2932] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } } }
I20240502 08:18:19.550136 19126 raft_consensus.cc:2932] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } } }
I20240502 08:18:19.551409 19278 raft_consensus.cc:2932] T 47ec01cade6e41d1a0ff6c8929ee8155 P 7d576eb536a94138b805dc5ef8b0800c [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } } }
I20240502 08:18:19.556027 19020 catalog_manager.cc:5484] T 47ec01cade6e41d1a0ff6c8929ee8155 P 86029a0bbf0b4f77b9c0e96cf121e154 reported cstate change: config changed from index -1 to 27, NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) added. New cstate: current_term: 2 leader_uuid: "287e8970c9524c998a437fbb3349fb60" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } } }
I20240502 08:18:19.557015 19202 consensus_queue.cc:237] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } }
I20240502 08:18:19.559756 19278 raft_consensus.cc:1261] T d3591d08cc604b349553a185225ef943 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 08:18:19.560212 19896 consensus_queue.cc:1035] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 08:18:19.563186 19854 raft_consensus.cc:2932] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } } }
I20240502 08:18:19.563730 19126 raft_consensus.cc:1261] T d3591d08cc604b349553a185225ef943 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 08:18:19.568089 19828 consensus_queue.cc:1035] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 08:18:19.568672 19278 raft_consensus.cc:2932] T d3591d08cc604b349553a185225ef943 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } } }
I20240502 08:18:19.573472 19018 catalog_manager.cc:5484] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 reported cstate change: config changed from index -1 to 26, NON_VOTER 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) added. New cstate: current_term: 1 leader_uuid: "287e8970c9524c998a437fbb3349fb60" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:19.576311 19278 consensus_queue.cc:237] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } }
W20240502 08:18:19.578673 19168 consensus_peers.cc:485] T d3591d08cc604b349553a185225ef943 P 287e8970c9524c998a437fbb3349fb60 -> Peer 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6:37361): Couldn't send request to peer 49a329b1dd144199a81b68a046cf8a4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3591d08cc604b349553a185225ef943. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:19.581414 19690 raft_consensus.cc:1261] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d [term 3 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 08:18:19.581414 19202 raft_consensus.cc:1261] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Refusing update from remote peer 7d576eb536a94138b805dc5ef8b0800c: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 08:18:19.582055 19785 consensus_queue.cc:1035] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 08:18:19.582365 19865 consensus_queue.cc:1035] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 08:18:19.582917 19126 raft_consensus.cc:2932] T d3591d08cc604b349553a185225ef943 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } } }
I20240502 08:18:19.593241 19892 raft_consensus.cc:2932] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 29 to 30, NON_VOTER e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } } }
I20240502 08:18:19.593947 19690 raft_consensus.cc:2932] T db356e6346d948938202e72df77e8199 P 49a329b1dd144199a81b68a046cf8a4d [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, NON_VOTER e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } } }
I20240502 08:18:19.594679 19202 raft_consensus.cc:2932] T db356e6346d948938202e72df77e8199 P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, NON_VOTER e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } } }
W20240502 08:18:19.599200 19244 consensus_peers.cc:485] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c -> Peer e6313d7fc95e46e88fb241f192368b1e (127.15.81.5:41865): Couldn't send request to peer e6313d7fc95e46e88fb241f192368b1e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db356e6346d948938202e72df77e8199. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:19.599575 19126 consensus_queue.cc:237] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } }
I20240502 08:18:19.599673 19019 catalog_manager.cc:5484] T db356e6346d948938202e72df77e8199 P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: config changed from index 29 to 30, NON_VOTER e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) added. New cstate: current_term: 3 leader_uuid: "7d576eb536a94138b805dc5ef8b0800c" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:19.609001 19202 raft_consensus.cc:1261] T 5990222a54794361819bebe9e51582e5 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
W20240502 08:18:19.609090 19093 consensus_peers.cc:485] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 -> Peer e6313d7fc95e46e88fb241f192368b1e (127.15.81.5:41865): Couldn't send request to peer e6313d7fc95e46e88fb241f192368b1e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5990222a54794361819bebe9e51582e5. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:19.609508 19913 consensus_queue.cc:1035] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 08:18:19.609643 19278 raft_consensus.cc:1261] T 5990222a54794361819bebe9e51582e5 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Refusing update from remote peer 86029a0bbf0b4f77b9c0e96cf121e154: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 08:18:19.610148 19913 consensus_queue.cc:1035] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 08:18:19.612248 19850 raft_consensus.cc:2932] T 5990222a54794361819bebe9e51582e5 P 86029a0bbf0b4f77b9c0e96cf121e154 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } } }
I20240502 08:18:19.612787 19202 raft_consensus.cc:2932] T 5990222a54794361819bebe9e51582e5 P 287e8970c9524c998a437fbb3349fb60 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } } }
I20240502 08:18:19.616226 19278 raft_consensus.cc:2932] T 5990222a54794361819bebe9e51582e5 P 7d576eb536a94138b805dc5ef8b0800c [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } } }
I20240502 08:18:19.617774 19202 consensus_queue.cc:237] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 3.25, Last appended by leader: 24, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } }
I20240502 08:18:19.617575 19020 catalog_manager.cc:5484] T 5990222a54794361819bebe9e51582e5 P 287e8970c9524c998a437fbb3349fb60 reported cstate change: config changed from index -1 to 25, NON_VOTER e6313d7fc95e46e88fb241f192368b1e (127.15.81.5) added. New cstate: current_term: 1 leader_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } attrs { replace: true } } peers { permanent_uuid: "e6313d7fc95e46e88fb241f192368b1e" member_type: NON_VOTER last_known_addr { host: "127.15.81.5" port: 41865 } attrs { promote: true } } }
I20240502 08:18:19.622103 19126 raft_consensus.cc:1261] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 3 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 3 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
W20240502 08:18:19.622268 19169 consensus_peers.cc:485] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 -> Peer 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4:39603): Couldn't send request to peer 4fa05fa6be33496798a23398a84ffbc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a21d56d23364d6f802d89fbc035c02f. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:19.622395 19279 raft_consensus.cc:1261] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 3 FOLLOWER]: Refusing update from remote peer 287e8970c9524c998a437fbb3349fb60: Log matching property violated. Preceding OpId in replica: term: 3 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
I20240502 08:18:19.622892 19854 consensus_queue.cc:1035] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 08:18:19.623258 19828 consensus_queue.cc:1035] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 08:18:19.625685 19854 raft_consensus.cc:2932] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 [term 3 LEADER]: Committing config change with OpId 3.26: config changed from index -1 to 26, NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } } }
I20240502 08:18:19.626708 19122 raft_consensus.cc:2932] T 6a21d56d23364d6f802d89fbc035c02f P 86029a0bbf0b4f77b9c0e96cf121e154 [term 3 FOLLOWER]: Committing config change with OpId 3.26: config changed from index -1 to 26, NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } } }
I20240502 08:18:19.628082 19278 raft_consensus.cc:2932] T 6a21d56d23364d6f802d89fbc035c02f P 7d576eb536a94138b805dc5ef8b0800c [term 3 FOLLOWER]: Committing config change with OpId 3.26: config changed from index -1 to 26, NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } } }
I20240502 08:18:19.631098 19018 catalog_manager.cc:5484] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60 reported cstate change: config changed from index -1 to 26, NON_VOTER 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4) added. New cstate: current_term: 3 leader_uuid: "287e8970c9524c998a437fbb3349fb60" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "86029a0bbf0b4f77b9c0e96cf121e154" member_type: VOTER last_known_addr { host: "127.15.81.1" port: 36079 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: NON_VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:19.639792 19540 consensus_queue.cc:237] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } }
I20240502 08:18:19.641698 19278 raft_consensus.cc:1261] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 3 FOLLOWER]: Refusing update from remote peer 4fa05fa6be33496798a23398a84ffbc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 08:18:19.642241 19202 raft_consensus.cc:1261] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Refusing update from remote peer 4fa05fa6be33496798a23398a84ffbc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 08:18:19.642375 19858 consensus_queue.cc:1035] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 08:18:19.642786 19893 consensus_queue.cc:1035] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 08:18:19.644961 19857 raft_consensus.cc:2932] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 29 to 31, NON_VOTER 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } } }
I20240502 08:18:19.645576 19278 raft_consensus.cc:2932] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, NON_VOTER 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } } }
I20240502 08:18:19.645623 19202 raft_consensus.cc:2932] T 2f640265eb3f4b10a0675e3b08b11680 P 287e8970c9524c998a437fbb3349fb60 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, NON_VOTER 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } } }
W20240502 08:18:19.649577 19505 consensus_peers.cc:485] T 2f640265eb3f4b10a0675e3b08b11680 P 4fa05fa6be33496798a23398a84ffbc9 -> Peer 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6:37361): Couldn't send request to peer 49a329b1dd144199a81b68a046cf8a4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f640265eb3f4b10a0675e3b08b11680. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:19.651299 19019 catalog_manager.cc:5484] T 2f640265eb3f4b10a0675e3b08b11680 P 7d576eb536a94138b805dc5ef8b0800c reported cstate change: config changed from index 29 to 31, NON_VOTER 49a329b1dd144199a81b68a046cf8a4d (127.15.81.6) added. New cstate: current_term: 3 leader_uuid: "4fa05fa6be33496798a23398a84ffbc9" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "287e8970c9524c998a437fbb3349fb60" member_type: VOTER last_known_addr { host: "127.15.81.2" port: 38313 } attrs { replace: true } } peers { permanent_uuid: "7d576eb536a94138b805dc5ef8b0800c" member_type: VOTER last_known_addr { host: "127.15.81.3" port: 40169 } } peers { permanent_uuid: "4fa05fa6be33496798a23398a84ffbc9" member_type: VOTER last_known_addr { host: "127.15.81.4" port: 39603 } attrs { promote: false } } peers { permanent_uuid: "49a329b1dd144199a81b68a046cf8a4d" member_type: NON_VOTER last_known_addr { host: "127.15.81.6" port: 37361 } attrs { promote: true } } }
I20240502 08:18:19.981446 19921 ts_tablet_manager.cc:902] T 47ec01cade6e41d1a0ff6c8929ee8155 P 4fa05fa6be33496798a23398a84ffbc9: Initiating tablet copy from peer 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:19.983070 19921 tablet_copy_client.cc:308] T 47ec01cade6e41d1a0ff6c8929ee8155 P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Beginning tablet copy session from remote peer at address 127.15.81.2:38313
I20240502 08:18:19.983808 19214 tablet_copy_service.cc:140] P 287e8970c9524c998a437fbb3349fb60: Received BeginTabletCopySession request for tablet 47ec01cade6e41d1a0ff6c8929ee8155 from peer 4fa05fa6be33496798a23398a84ffbc9 ({username='slave'} at 127.0.0.1:53080)
I20240502 08:18:19.983958 19214 tablet_copy_service.cc:161] P 287e8970c9524c998a437fbb3349fb60: Beginning new tablet copy session on tablet 47ec01cade6e41d1a0ff6c8929ee8155 from peer 4fa05fa6be33496798a23398a84ffbc9 at {username='slave'} at 127.0.0.1:53080: session id = 4fa05fa6be33496798a23398a84ffbc9-47ec01cade6e41d1a0ff6c8929ee8155
I20240502 08:18:19.986006 19214 tablet_copy_source_session.cc:205] T 47ec01cade6e41d1a0ff6c8929ee8155 P 287e8970c9524c998a437fbb3349fb60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:19.987211 19921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ec01cade6e41d1a0ff6c8929ee8155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.993367 19921 tablet_copy_client.cc:783] T 47ec01cade6e41d1a0ff6c8929ee8155 P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:19.993723 19921 tablet_copy_client.cc:647] T 47ec01cade6e41d1a0ff6c8929ee8155 P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:19.996130 19925 tablet_copy_client.cc:1002] T 47ec01cade6e41d1a0ff6c8929ee8155 P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Injecting latency into file download: 2000
I20240502 08:18:20.028146 19926 ts_tablet_manager.cc:902] T 6a21d56d23364d6f802d89fbc035c02f P 4fa05fa6be33496798a23398a84ffbc9: Initiating tablet copy from peer 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
I20240502 08:18:20.030426 19926 tablet_copy_client.cc:308] T 6a21d56d23364d6f802d89fbc035c02f P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Beginning tablet copy session from remote peer at address 127.15.81.2:38313
I20240502 08:18:20.031289 19214 tablet_copy_service.cc:140] P 287e8970c9524c998a437fbb3349fb60: Received BeginTabletCopySession request for tablet 6a21d56d23364d6f802d89fbc035c02f from peer 4fa05fa6be33496798a23398a84ffbc9 ({username='slave'} at 127.0.0.1:53080)
I20240502 08:18:20.031479 19214 tablet_copy_service.cc:161] P 287e8970c9524c998a437fbb3349fb60: Beginning new tablet copy session on tablet 6a21d56d23364d6f802d89fbc035c02f from peer 4fa05fa6be33496798a23398a84ffbc9 at {username='slave'} at 127.0.0.1:53080: session id = 4fa05fa6be33496798a23398a84ffbc9-6a21d56d23364d6f802d89fbc035c02f
I20240502 08:18:20.034757 19214 tablet_copy_source_session.cc:205] T 6a21d56d23364d6f802d89fbc035c02f P 287e8970c9524c998a437fbb3349fb60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:20.036532 19931 ts_tablet_manager.cc:902] T 5990222a54794361819bebe9e51582e5 P e6313d7fc95e46e88fb241f192368b1e: Initiating tablet copy from peer 86029a0bbf0b4f77b9c0e96cf121e154 (127.15.81.1:36079)
I20240502 08:18:20.036563 19926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a21d56d23364d6f802d89fbc035c02f. 1 dirs total, 0 dirs full, 0 dirs failed
F20240502 08:18:20.037163 19931 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-020165b5b7c743039aa2b3a5a818d17b.queue_time_us, tablet-download-pool-020165b5b7c743039aa2b3a5a818d17b.queue_time_us) (tablet-download-pool-020165b5b7c743039aa2b3a5a818d17b.run_cpu_time_us, tablet-download-pool-020165b5b7c743039aa2b3a5a818d17b.run_cpu_time_us) (tablet-download-pool-020165b5b7c743039aa2b3a5a818d17b.run_wall_time_us, tablet-download-pool-020165b5b7c743039aa2b3a5a818d17b.run_wall_time_us) (tablet-download-pool-2f640265eb3f4b10a0675e3b08b11680.queue_time_us, tablet-download-pool-2f640265eb3f4b10a0675e3b08b11680.queue_time_us) (tablet-download-pool-2f640265eb3f4b10a0675e3b08b11680.run_cpu_time_us, tablet-download-pool-2f640265eb3f4b10a0675e3b08b11680.run_cpu_time_us) (tablet-download-pool-2f640265eb3f4b10a0675e3b08b11680.run_wall_time_us, tablet-download-pool-2f640265eb3f4b10a0675e3b08b11680.run_wall_time_us) (tablet-download-pool-47ec01cade6e41d1a0ff6c8929ee8155.queue_time_us, tablet-download-pool-47ec01cade6e41d1a0ff6c8929ee8155.queue_time_us) (tablet-download-pool-47ec01cade6e41d1a0ff6c8929ee8155.run_cpu_time_us, tablet-download-pool-47ec01cade6e41d1a0ff6c8929ee8155.run_cpu_time_us) (tablet-download-pool-47ec01cade6e41d1a0ff6c8929ee8155.run_wall_time_us, tablet-download-pool-47ec01cade6e41d1a0ff6c8929ee8155.run_wall_time_us) (tablet-download-pool-4febdd8875b64cc1a156527854a1167a.queue_time_us, tablet-download-pool-4febdd8875b64cc1a156527854a1167a.queue_time_us) (tablet-download-pool-4febdd8875b64cc1a156527854a1167a.run_cpu_time_us, tablet-download-pool-4febdd8875b64cc1a156527854a1167a.run_cpu_time_us) (tablet-download-pool-4febdd8875b64cc1a156527854a1167a.run_wall_time_us, tablet-download-pool-4febdd8875b64cc1a156527854a1167a.run_wall_time_us) (tablet-download-pool-5863fbeaebcb43229cf2e665a9a95ca1.queue_time_us, tablet-download-pool-5863fbeaebcb43229cf2e665a9a95ca1.queue_time_us) (tablet-download-pool-5863fbeaebcb43229cf2e665a9a95ca1.run_cpu_time_us, tablet-download-pool-5863fbeaebcb43229cf2e665a9a95ca1.run_cpu_time_us) (tablet-download-pool-5863fbeaebcb43229cf2e665a9a95ca1.run_wall_time_us, tablet-download-pool-5863fbeaebcb43229cf2e665a9a95ca1.run_wall_time_us) (tablet-download-pool-5990222a54794361819bebe9e51582e5.queue_time_us, tablet-download-pool-5990222a54794361819bebe9e51582e5.queue_time_us) (tablet-download-pool-5fa05fba592744fdbfb354d227626c26.queue_time_us, tablet-download-pool-5fa05fba592744fdbfb354d227626c26.queue_time_us) (tablet-download-pool-5fa05fba592744fdbfb354d227626c26.run_cpu_time_us, tablet-download-pool-5fa05fba592744fdbfb354d227626c26.run_cpu_time_us) (tablet-download-pool-5fa05fba592744fdbfb354d227626c26.run_wall_time_us, tablet-download-pool-5fa05fba592744fdbfb354d227626c26.run_wall_time_us) (tablet-download-pool-6a21d56d23364d6f802d89fbc035c02f.queue_time_us, tablet-download-pool-6a21d56d23364d6f802d89fbc035c02f.queue_time_us) (tablet-download-pool-6a21d56d23364d6f802d89fbc035c02f.run_cpu_time_us, tablet-download-pool-6a21d56d23364d6f802d89fbc035c02f.run_cpu_time_us) (tablet-download-pool-6a21d56d23364d6f802d89fbc035c02f.run_wall_time_us, tablet-download-pool-6a21d56d23364d6f802d89fbc035c02f.run_wall_time_us) (tablet-download-pool-8fa1524bb0a749f9998928d10f4adea6.queue_time_us, tablet-download-pool-8fa1524bb0a749f9998928d10f4adea6.queue_time_us) (tablet-download-pool-8fa1524bb0a749f9998928d10f4adea6.run_cpu_time_us, tablet-download-pool-8fa1524bb0a749f9998928d10f4adea6.run_cpu_time_us) (tablet-download-pool-8fa1524bb0a749f9998928d10f4adea6.run_wall_time_us, tablet-download-pool-8fa1524bb0a749f9998928d10f4adea6.run_wall_time_us) (tablet-download-pool-96eecf3a17634ff39abdce6c99f4eaa6.queue_time_us, tablet-download-pool-96eecf3a17634ff39abdce6c99f4eaa6.queue_time_us) (tablet-download-pool-96eecf3a17634ff39abdce6c99f4eaa6.run_cpu_time_us, tablet-download-pool-96eecf3a17634ff39abdce6c99f4eaa6.run_cpu_time_us) (tablet-download-pool-96eecf3a17634ff39abdce6c99f4eaa6.run_wall_time_us, tablet-download-pool-96eecf3a17634ff39abdce6c99f4eaa6.run_wall_time_us) (tablet-download-pool-b48005466a39437b937e2d09c1826d27.queue_time_us, tablet-download-pool-b48005466a39437b937e2d09c1826d27.queue_time_us) (tablet-download-pool-b48005466a39437b937e2d09c1826d27.run_cpu_time_us, tablet-download-pool-b48005466a39437b937e2d09c1826d27.run_cpu_time_us) (tablet-download-pool-b48005466a39437b937e2d09c1826d27.run_wall_time_us, tablet-download-pool-b48005466a39437b937e2d09c1826d27.run_wall_time_us) (tablet-download-pool-c4003407906b496b89662cfaf89d66be.queue_time_us, tablet-download-pool-c4003407906b496b89662cfaf89d66be.queue_time_us) (tablet-download-pool-c4003407906b496b89662cfaf89d66be.run_cpu_time_us, tablet-download-pool-c4003407906b496b89662cfaf89d66be.run_cpu_time_us) (tablet-download-pool-c4003407906b496b89662cfaf89d66be.run_wall_time_us, tablet-download-pool-c4003407906b496b89662cfaf89d66be.run_wall_time_us) (tablet-download-pool-db356e6346d948938202e72df77e8199.queue_time_us, tablet-download-pool-db356e6346d948938202e72df77e8199.queue_time_us) (tablet-download-pool-db356e6346d948938202e72df77e8199.run_cpu_time_us, tablet-download-pool-db356e6346d948938202e72df77e8199.run_cpu_time_us) (tablet-download-pool-db356e6346d948938202e72df77e8199.run_wall_time_us, tablet-download-pool-db356e6346d948938202e72df77e8199.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714637900 (unix time) try "date -d @1714637900" if you are using GNU date ***
I20240502 08:18:20.042678 19926 tablet_copy_client.cc:783] T 6a21d56d23364d6f802d89fbc035c02f P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:20.043056 19926 tablet_copy_client.cc:647] T 6a21d56d23364d6f802d89fbc035c02f P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:20.045344 19930 tablet_copy_client.cc:1002] T 6a21d56d23364d6f802d89fbc035c02f P 4fa05fa6be33496798a23398a84ffbc9: tablet copy: Injecting latency into file download: 2000
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800003d44) received by PID 15684 (TID 0x7fa9856c5700) from PID 15684; stack trace: ***
@ 0x7faa4b6fc980 (unknown) at ??:0
@ 0x7faa40a49fb7 gsignal at ??:0
@ 0x7faa40a4b921 abort at ??:0
@ 0x7faa434fedcd google::LogMessage::Fail() at ??:0
@ 0x7faa43502b93 google::LogMessage::SendToLog() at ??:0
@ 0x7faa434fe7cc google::LogMessage::Flush() at ??:0
@ 0x7faa434fff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7faa44b7b742 kudu::TraceMetrics::InternName() at ??:0
I20240502 08:18:20.092443 19938 ts_tablet_manager.cc:902] T 2f640265eb3f4b10a0675e3b08b11680 P 49a329b1dd144199a81b68a046cf8a4d: Initiating tablet copy from peer 4fa05fa6be33496798a23398a84ffbc9 (127.15.81.4:39603)
@ 0x7faa44b47f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 08:18:20.101523 19942 ts_tablet_manager.cc:902] T d3591d08cc604b349553a185225ef943 P 49a329b1dd144199a81b68a046cf8a4d: Initiating tablet copy from peer 287e8970c9524c998a437fbb3349fb60 (127.15.81.2:38313)
@ 0x7faa44b41cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7faa5400d8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7faa54025daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7faa54150480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7faa54188a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7faa5416b981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7faa5416b70e std::_Function_handler<>::_M_invoke() at ??:0
I20240502 08:18:20.145503 19949 ts_tablet_manager.cc:902] T db356e6346d948938202e72df77e8199 P e6313d7fc95e46e88fb241f192368b1e: Initiating tablet copy from peer 7d576eb536a94138b805dc5ef8b0800c (127.15.81.3:40169)
@ 0x7faa5900d98d std::function<>::operator()() at ??:0
@ 0x7faa44b4bb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7faa44b4f19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7faa44b4f019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7faa5900d98d std::function<>::operator()() at ??:0
@ 0x7faa44b2a01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7faa4b6f16db start_thread at ??:0
@ 0x7faa40b2c71f clone at ??:0