[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 14:13:43.311172 15657 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.74.126:40475,127.15.74.125:33081,127.15.74.124:38935
I20240502 14:13:43.313051 15657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:43.314249 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:43.326457 15663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:43.327003 15665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:43.327876 15662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:43.328734 15657 server_base.cc:1034] running on GCE node
I20240502 14:13:43.329568 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:43.329759 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:43.329849 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659223329828 us; error 0 us; skew 500 ppm
I20240502 14:13:43.330184 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:43.333222 15657 webserver.cc:460] Webserver started at http://127.15.74.126:42403/ using document root <none> and password file <none>
I20240502 14:13:43.333997 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:43.334105 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:43.334468 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:43.337922 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-0-root/instance:
uuid: "3347f91e1d9143c38b128507a98008c5"
format_stamp: "Formatted at 2024-05-02 14:13:43 on dist-test-slave-wbm6"
I20240502 14:13:43.343786 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20240502 14:13:43.347119 15670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:43.348649 15657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:13:43.348837 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-0-root
uuid: "3347f91e1d9143c38b128507a98008c5"
format_stamp: "Formatted at 2024-05-02 14:13:43 on dist-test-slave-wbm6"
I20240502 14:13:43.349025 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:43.385006 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:43.385949 15657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:43.386240 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:43.404891 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.126:40475
I20240502 14:13:43.404924 15721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.126:40475 every 8 connection(s)
I20240502 14:13:43.407037 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:43.410651 15724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:43.410786 15722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:13:43.411249 15725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:43.412091 15727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:43.412935 15657 server_base.cc:1034] running on GCE node
I20240502 14:13:43.413245 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:43.413316 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:43.413393 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659223413354 us; error 0 us; skew 500 ppm
I20240502 14:13:43.413578 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:43.415030 15657 webserver.cc:460] Webserver started at http://127.15.74.125:40937/ using document root <none> and password file <none>
I20240502 14:13:43.415275 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:43.415367 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:43.415542 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:43.416193 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-1-root/instance:
uuid: "71adf07ec4984de784bb91f1855e03ca"
format_stamp: "Formatted at 2024-05-02 14:13:43 on dist-test-slave-wbm6"
I20240502 14:13:43.419018 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:13:43.420814 15732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:43.421195 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 14:13:43.421295 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-1-root
uuid: "71adf07ec4984de784bb91f1855e03ca"
format_stamp: "Formatted at 2024-05-02 14:13:43 on dist-test-slave-wbm6"
I20240502 14:13:43.421439 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:43.421993 15722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:13:43.432026 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:43.432792 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:43.439496 15722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
W20240502 14:13:43.441144 15673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.74.125:33081: connect: Connection refused (error 111)
W20240502 14:13:43.443017 15722 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.15.74.125:33081: Network error: Client connection negotiation failed: client connection to 127.15.74.125:33081: connect: Connection refused (error 111)
I20240502 14:13:43.452118 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.125:33081
I20240502 14:13:43.452167 15786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.125:33081 every 8 connection(s)
I20240502 14:13:43.453708 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:13:43.453759 15787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:13:43.457705 15790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:43.458137 15787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } has no permanent_uuid. Determining permanent_uuid...
W20240502 14:13:43.458478 15789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:43.459412 15657 server_base.cc:1034] running on GCE node
W20240502 14:13:43.459810 15792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:43.460232 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:43.460327 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:43.460420 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659223460380 us; error 0 us; skew 500 ppm
I20240502 14:13:43.460594 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:43.462066 15657 webserver.cc:460] Webserver started at http://127.15.74.124:37731/ using document root <none> and password file <none>
I20240502 14:13:43.462332 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:43.462442 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:43.462575 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:43.463397 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-2-root/instance:
uuid: "562dbd4b638a4e318bf48f05660e401e"
format_stamp: "Formatted at 2024-05-02 14:13:43 on dist-test-slave-wbm6"
I20240502 14:13:43.464989 15787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:13:43.468279 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20240502 14:13:43.470474 15799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:43.471006 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 14:13:43.471151 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-2-root
uuid: "562dbd4b638a4e318bf48f05660e401e"
format_stamp: "Formatted at 2024-05-02 14:13:43 on dist-test-slave-wbm6"
I20240502 14:13:43.471346 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:43.472544 15787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } has no permanent_uuid. Determining permanent_uuid...
W20240502 14:13:43.474958 15787 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.15.74.124:38935: Network error: Client connection negotiation failed: client connection to 127.15.74.124:38935: connect: Connection refused (error 111)
I20240502 14:13:43.487991 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:43.488651 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:43.492413 15722 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } attempt: 1
I20240502 14:13:43.500567 15722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } has no permanent_uuid. Determining permanent_uuid...
W20240502 14:13:43.502925 15722 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.15.74.124:38935: Network error: Client connection negotiation failed: client connection to 127.15.74.124:38935: connect: Connection refused (error 111)
I20240502 14:13:43.508028 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.124:38935
I20240502 14:13:43.508091 15851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.124:38935 every 8 connection(s)
I20240502 14:13:43.509171 15657 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:13:43.509709 15852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:43.512200 15852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:13:43.518316 15852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:13:43.523712 15852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:13:43.527154 15787 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } attempt: 1
I20240502 14:13:43.535130 15852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e: Bootstrap starting.
I20240502 14:13:43.535995 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca: Bootstrap starting.
I20240502 14:13:43.538786 15852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:43.538791 15787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:43.540139 15852 log.cc:824] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:43.542593 15852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e: No bootstrap required, opened a new log
I20240502 14:13:43.542598 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca: No bootstrap required, opened a new log
I20240502 14:13:43.546196 15722 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } attempt: 1
I20240502 14:13:43.551733 15852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } }
I20240502 14:13:43.551709 15787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } }
I20240502 14:13:43.552165 15852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:43.552170 15787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:43.552345 15852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562dbd4b638a4e318bf48f05660e401e, State: Initialized, Role: FOLLOWER
I20240502 14:13:43.552348 15787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71adf07ec4984de784bb91f1855e03ca, State: Initialized, Role: FOLLOWER
I20240502 14:13:43.553375 15852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } }
I20240502 14:13:43.553929 15787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } }
I20240502 14:13:43.560714 15722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5: Bootstrap starting.
I20240502 14:13:43.562935 15787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:13:43.563167 15722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:43.564263 15852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:13:43.564090 15859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } } }
I20240502 14:13:43.564481 15859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:13:43.564695 15860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } } }
I20240502 14:13:43.565583 15860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:13:43.572185 15722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5: No bootstrap required, opened a new log
I20240502 14:13:43.573494 15722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } }
I20240502 14:13:43.573803 15722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:43.573906 15722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3347f91e1d9143c38b128507a98008c5, State: Initialized, Role: FOLLOWER
I20240502 14:13:43.574311 15722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } }
I20240502 14:13:43.579604 15722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:13:43.578661 15865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } } }
I20240502 14:13:43.581486 15865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:13:43.615363 15888 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240502 14:13:43.616902 15657 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240502 14:13:43.617866 15657 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240502 14:13:43.619041 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:43.623976 15893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 14:13:43.624279 15893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 14:13:43.624768 15895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 14:13:43.624873 15895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 14:13:43.627334 15897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:43.629460 15898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:43.633322 15657 server_base.cc:1034] running on GCE node
W20240502 14:13:43.634171 15900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:43.634599 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:43.634713 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:43.634779 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659223634759 us; error 0 us; skew 500 ppm
I20240502 14:13:43.634989 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:43.636732 15657 webserver.cc:460] Webserver started at http://127.15.74.65:45725/ using document root <none> and password file <none>
I20240502 14:13:43.637037 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:43.637146 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:43.637306 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:43.638262 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-0-root/instance:
uuid: "e5a8d9bf10df4cc0b22899c97b261a3b"
format_stamp: "Formatted at 2024-05-02 14:13:43 on dist-test-slave-wbm6"
W20240502 14:13:43.638445 15676 tablet.cc:2369] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:13:43.642130 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:13:43.644591 15905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:43.645059 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:13:43.645201 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-0-root
uuid: "e5a8d9bf10df4cc0b22899c97b261a3b"
format_stamp: "Formatted at 2024-05-02 14:13:43 on dist-test-slave-wbm6"
I20240502 14:13:43.645356 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:43.659961 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:43.660800 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:43.661916 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:43.664013 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:43.664173 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:43.664322 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:43.664419 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:43.691138 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.65:44019
I20240502 14:13:43.691215 15967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.65:44019 every 8 connection(s)
I20240502 14:13:43.713802 15969 heartbeater.cc:347] Connected to a master server at 127.15.74.126:40475
I20240502 14:13:43.714150 15969 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:43.715337 15969 heartbeater.cc:511] Master 127.15.74.126:40475 requested a full tablet report, sending...
I20240502 14:13:43.716400 15968 heartbeater.cc:347] Connected to a master server at 127.15.74.124:38935
I20240502 14:13:43.716549 15968 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:43.717217 15968 heartbeater.cc:511] Master 127.15.74.124:38935 requested a full tablet report, sending...
I20240502 14:13:43.718425 15687 ts_manager.cc:194] Registered new tserver with Master: e5a8d9bf10df4cc0b22899c97b261a3b (127.15.74.65:44019)
I20240502 14:13:43.718737 15817 ts_manager.cc:194] Registered new tserver with Master: e5a8d9bf10df4cc0b22899c97b261a3b (127.15.74.65:44019)
I20240502 14:13:43.721349 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:43.725602 15974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:43.730235 15970 heartbeater.cc:347] Connected to a master server at 127.15.74.125:33081
I20240502 14:13:43.730540 15970 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:43.731840 15970 heartbeater.cc:511] Master 127.15.74.125:33081 requested a full tablet report, sending...
I20240502 14:13:43.734412 15751 ts_manager.cc:194] Registered new tserver with Master: e5a8d9bf10df4cc0b22899c97b261a3b (127.15.74.65:44019)
W20240502 14:13:43.735371 15975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:43.738862 15978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:43.739728 15657 server_base.cc:1034] running on GCE node
I20240502 14:13:43.740043 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:43.740118 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:43.740204 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659223740158 us; error 0 us; skew 500 ppm
I20240502 14:13:43.740368 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:43.741993 15657 webserver.cc:460] Webserver started at http://127.15.74.66:32829/ using document root <none> and password file <none>
I20240502 14:13:43.742244 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:43.742353 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:43.742506 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:43.743183 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-1-root/instance:
uuid: "474981443c8845e59a9ab2348bf59b6d"
format_stamp: "Formatted at 2024-05-02 14:13:43 on dist-test-slave-wbm6"
I20240502 14:13:43.746160 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240502 14:13:43.746968 15865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:43.747288 15865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } }
I20240502 14:13:43.748369 15982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:43.748782 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:13:43.748905 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-1-root
uuid: "474981443c8845e59a9ab2348bf59b6d"
format_stamp: "Formatted at 2024-05-02 14:13:43 on dist-test-slave-wbm6"
I20240502 14:13:43.749042 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:43.749565 15865 leader_election.cc:288] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71adf07ec4984de784bb91f1855e03ca (127.15.74.125:33081), 562dbd4b638a4e318bf48f05660e401e (127.15.74.124:38935)
I20240502 14:13:43.750449 15762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3347f91e1d9143c38b128507a98008c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71adf07ec4984de784bb91f1855e03ca" is_pre_election: true
I20240502 14:13:43.751015 15762 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3347f91e1d9143c38b128507a98008c5 in term 0.
I20240502 14:13:43.750939 15827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3347f91e1d9143c38b128507a98008c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562dbd4b638a4e318bf48f05660e401e" is_pre_election: true
I20240502 14:13:43.751345 15827 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3347f91e1d9143c38b128507a98008c5 in term 0.
I20240502 14:13:43.751819 15673 leader_election.cc:302] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3347f91e1d9143c38b128507a98008c5, 71adf07ec4984de784bb91f1855e03ca; no voters:
I20240502 14:13:43.752491 15865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:13:43.752661 15865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:43.752769 15865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:43.756073 15865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } }
I20240502 14:13:43.756788 15865 leader_election.cc:288] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [CANDIDATE]: Term 1 election: Requested vote from peers 71adf07ec4984de784bb91f1855e03ca (127.15.74.125:33081), 562dbd4b638a4e318bf48f05660e401e (127.15.74.124:38935)
I20240502 14:13:43.757381 15827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3347f91e1d9143c38b128507a98008c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562dbd4b638a4e318bf48f05660e401e"
I20240502 14:13:43.757520 15762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3347f91e1d9143c38b128507a98008c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71adf07ec4984de784bb91f1855e03ca"
I20240502 14:13:43.757586 15827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:43.757692 15762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:43.760874 15762 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3347f91e1d9143c38b128507a98008c5 in term 1.
I20240502 14:13:43.761025 15827 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3347f91e1d9143c38b128507a98008c5 in term 1.
I20240502 14:13:43.761323 15673 leader_election.cc:302] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3347f91e1d9143c38b128507a98008c5, 71adf07ec4984de784bb91f1855e03ca; no voters:
I20240502 14:13:43.761744 15865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:43.762239 15865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [term 1 LEADER]: Becoming Leader. State: Replica: 3347f91e1d9143c38b128507a98008c5, State: Running, Role: LEADER
I20240502 14:13:43.762912 15865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } }
I20240502 14:13:43.767024 15987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3347f91e1d9143c38b128507a98008c5. Latest consensus state: current_term: 1 leader_uuid: "3347f91e1d9143c38b128507a98008c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } } }
I20240502 14:13:43.767361 15987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:43.771850 15989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:13:43.774183 15989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:13:43.783834 15827 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [term 1 FOLLOWER]: Refusing update from remote peer 3347f91e1d9143c38b128507a98008c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:13:43.783834 15762 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [term 1 FOLLOWER]: Refusing update from remote peer 3347f91e1d9143c38b128507a98008c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:13:43.784725 15987 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:43.785180 15865 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:43.785738 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:43.786566 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:43.787374 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:43.792165 15859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3347f91e1d9143c38b128507a98008c5. Latest consensus state: current_term: 1 leader_uuid: "3347f91e1d9143c38b128507a98008c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } } }
I20240502 14:13:43.792467 15859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:13:43.794219 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:43.794312 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:43.794428 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:43.794500 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:43.796224 15993 mvcc.cc:205] Tried to move back new op lower bound from 7023244180607066112 to 7023244180539392000. Current Snapshot: MvccSnapshot[applied={T|T < 7023244180607066112}]
I20240502 14:13:43.813879 15987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3347f91e1d9143c38b128507a98008c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } } }
I20240502 14:13:43.814302 15987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:43.816293 15859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3347f91e1d9143c38b128507a98008c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } } }
I20240502 14:13:43.816533 15859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:13:43.818611 15860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3347f91e1d9143c38b128507a98008c5. Latest consensus state: current_term: 1 leader_uuid: "3347f91e1d9143c38b128507a98008c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } } }
I20240502 14:13:43.819743 15860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:13:43.819051 15865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3347f91e1d9143c38b128507a98008c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } } }
I20240502 14:13:43.821873 15865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:43.827468 15989 catalog_manager.cc:1347] Generated new cluster ID: 4fbd4a1ae95c4b91a03d594c545e6544
I20240502 14:13:43.827617 15989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:13:43.837363 15860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3347f91e1d9143c38b128507a98008c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3347f91e1d9143c38b128507a98008c5" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 40475 } } peers { permanent_uuid: "71adf07ec4984de784bb91f1855e03ca" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 33081 } } peers { permanent_uuid: "562dbd4b638a4e318bf48f05660e401e" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 38935 } } }
I20240502 14:13:43.837702 15860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:13:43.860041 15989 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:13:43.861837 15989 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:13:43.876821 15989 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5: Generated new TSK 0
I20240502 14:13:43.878371 15989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:13:43.889109 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.66:40029
I20240502 14:13:43.889134 16056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.66:40029 every 8 connection(s)
I20240502 14:13:43.902951 16057 heartbeater.cc:347] Connected to a master server at 127.15.74.124:38935
I20240502 14:13:43.903129 16057 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:43.904063 16057 heartbeater.cc:511] Master 127.15.74.124:38935 requested a full tablet report, sending...
I20240502 14:13:43.907249 15817 ts_manager.cc:194] Registered new tserver with Master: 474981443c8845e59a9ab2348bf59b6d (127.15.74.66:40029)
I20240502 14:13:43.912097 16058 heartbeater.cc:347] Connected to a master server at 127.15.74.126:40475
I20240502 14:13:43.912313 16058 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:43.913116 16058 heartbeater.cc:511] Master 127.15.74.126:40475 requested a full tablet report, sending...
I20240502 14:13:43.914404 15687 ts_manager.cc:194] Registered new tserver with Master: 474981443c8845e59a9ab2348bf59b6d (127.15.74.66:40029)
I20240502 14:13:43.916285 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:13:43.917389 16059 heartbeater.cc:347] Connected to a master server at 127.15.74.125:33081
I20240502 14:13:43.917531 16059 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:43.917491 15687 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44282
I20240502 14:13:43.918139 16059 heartbeater.cc:511] Master 127.15.74.125:33081 requested a full tablet report, sending...
I20240502 14:13:43.919230 15751 ts_manager.cc:194] Registered new tserver with Master: 474981443c8845e59a9ab2348bf59b6d (127.15.74.66:40029)
W20240502 14:13:43.923368 16063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:43.924221 16064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:43.924882 16066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:43.925609 15657 server_base.cc:1034] running on GCE node
I20240502 14:13:43.925915 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:43.926000 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:43.926065 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659223926043 us; error 0 us; skew 500 ppm
I20240502 14:13:43.926251 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:43.927683 15657 webserver.cc:460] Webserver started at http://127.15.74.67:42767/ using document root <none> and password file <none>
I20240502 14:13:43.927938 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:43.928025 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:43.928175 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:43.928848 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-2-root/instance:
uuid: "e7db86a5a92747a7a3ed7090af751e3e"
format_stamp: "Formatted at 2024-05-02 14:13:43 on dist-test-slave-wbm6"
I20240502 14:13:43.931535 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 14:13:43.933359 16071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:43.933771 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:13:43.933889 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-2-root
uuid: "e7db86a5a92747a7a3ed7090af751e3e"
format_stamp: "Formatted at 2024-05-02 14:13:43 on dist-test-slave-wbm6"
I20240502 14:13:43.934031 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:43.943593 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:43.944280 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:43.944943 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:43.946098 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:43.946185 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:43.946290 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:43.946372 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:43.966481 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.67:46165
I20240502 14:13:43.966518 16133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.67:46165 every 8 connection(s)
I20240502 14:13:43.978899 16134 heartbeater.cc:347] Connected to a master server at 127.15.74.124:38935
I20240502 14:13:43.979074 16134 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:43.979827 16134 heartbeater.cc:511] Master 127.15.74.124:38935 requested a full tablet report, sending...
I20240502 14:13:43.981123 15817 ts_manager.cc:194] Registered new tserver with Master: e7db86a5a92747a7a3ed7090af751e3e (127.15.74.67:46165)
I20240502 14:13:43.989094 16135 heartbeater.cc:347] Connected to a master server at 127.15.74.126:40475
I20240502 14:13:43.989267 16135 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:43.989943 16135 heartbeater.cc:511] Master 127.15.74.126:40475 requested a full tablet report, sending...
I20240502 14:13:43.990602 16136 heartbeater.cc:347] Connected to a master server at 127.15.74.125:33081
I20240502 14:13:43.990762 16136 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:43.991120 15687 ts_manager.cc:194] Registered new tserver with Master: e7db86a5a92747a7a3ed7090af751e3e (127.15.74.67:46165)
I20240502 14:13:43.991345 16136 heartbeater.cc:511] Master 127.15.74.125:33081 requested a full tablet report, sending...
I20240502 14:13:43.992331 15687 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44296
I20240502 14:13:43.993247 15751 ts_manager.cc:194] Registered new tserver with Master: e7db86a5a92747a7a3ed7090af751e3e (127.15.74.67:46165)
I20240502 14:13:43.993686 15657 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.010500351s
I20240502 14:13:44.617506 15888 catalog_manager.cc:1259] Loaded cluster ID: 4fbd4a1ae95c4b91a03d594c545e6544
I20240502 14:13:44.617676 15888 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e: loading cluster ID for follower catalog manager: success
I20240502 14:13:44.619796 15888 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e: acquiring CA information for follower catalog manager: success
I20240502 14:13:44.621023 15888 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 14:13:44.626060 15895 catalog_manager.cc:1259] Loaded cluster ID: 4fbd4a1ae95c4b91a03d594c545e6544
I20240502 14:13:44.626137 15895 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca: loading cluster ID for follower catalog manager: success
I20240502 14:13:44.628037 15895 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca: acquiring CA information for follower catalog manager: success
I20240502 14:13:44.628988 15895 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 14:13:44.721670 15687 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44280
I20240502 14:13:44.920706 16058 heartbeater.cc:503] Master 127.15.74.126:40475 was elected leader, sending a full tablet report...
I20240502 14:13:44.993813 16135 heartbeater.cc:503] Master 127.15.74.126:40475 was elected leader, sending a full tablet report...
I20240502 14:13:45.616801 15891 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:13:45.722854 15969 heartbeater.cc:503] Master 127.15.74.126:40475 was elected leader, sending a full tablet report...
I20240502 14:13:46.050452 15657 test_util.cc:268] Using random seed: 907304826
I20240502 14:13:46.087977 15687 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44300:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 14:13:46.133159 16022 tablet_service.cc:1467] Processing CreateTablet for tablet 48368d9e05984de08f1f4f29d2e82986 (DEFAULT_TABLE table=test-workload [id=0f78aa45c2eb4c3481d6e7761704ae8a]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 14:13:46.134080 16022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48368d9e05984de08f1f4f29d2e82986. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:46.137703 16099 tablet_service.cc:1467] Processing CreateTablet for tablet 6686fa37c473414498b8405cfa421794 (DEFAULT_TABLE table=test-workload [id=0f78aa45c2eb4c3481d6e7761704ae8a]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 14:13:46.138263 16099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6686fa37c473414498b8405cfa421794. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:46.142505 16168 tablet_bootstrap.cc:492] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d: Bootstrap starting.
I20240502 14:13:46.144909 16169 tablet_bootstrap.cc:492] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e: Bootstrap starting.
I20240502 14:13:46.145064 15933 tablet_service.cc:1467] Processing CreateTablet for tablet 8a232292622e46318f8846708469e4f6 (DEFAULT_TABLE table=test-workload [id=0f78aa45c2eb4c3481d6e7761704ae8a]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 14:13:46.145591 15933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a232292622e46318f8846708469e4f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:46.146749 16169 tablet_bootstrap.cc:654] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:46.148510 15932 tablet_service.cc:1467] Processing CreateTablet for tablet bca696984f824a9a8f186139150b31ff (DEFAULT_TABLE table=test-workload [id=0f78aa45c2eb4c3481d6e7761704ae8a]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 14:13:46.148761 16169 tablet_bootstrap.cc:492] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e: No bootstrap required, opened a new log
I20240502 14:13:46.148938 16169 ts_tablet_manager.cc:1367] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:13:46.149039 15932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca696984f824a9a8f186139150b31ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:46.149726 16169 raft_consensus.cc:348] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7db86a5a92747a7a3ed7090af751e3e" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 46165 } }
I20240502 14:13:46.149916 16169 raft_consensus.cc:374] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:46.149998 16169 raft_consensus.cc:729] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7db86a5a92747a7a3ed7090af751e3e, State: Initialized, Role: FOLLOWER
I20240502 14:13:46.150264 16169 consensus_queue.cc:260] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7db86a5a92747a7a3ed7090af751e3e" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 46165 } }
I20240502 14:13:46.150440 16169 raft_consensus.cc:388] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:46.150540 16169 raft_consensus.cc:482] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:46.150684 16169 raft_consensus.cc:3037] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:46.153263 16171 tablet_bootstrap.cc:492] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b: Bootstrap starting.
I20240502 14:13:46.153059 16169 raft_consensus.cc:504] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7db86a5a92747a7a3ed7090af751e3e" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 46165 } }
I20240502 14:13:46.153702 16169 leader_election.cc:302] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7db86a5a92747a7a3ed7090af751e3e; no voters:
I20240502 14:13:46.154243 16169 leader_election.cc:288] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:46.154412 16172 raft_consensus.cc:2781] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:46.156102 16171 tablet_bootstrap.cc:654] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:46.156721 16168 tablet_bootstrap.cc:654] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:46.162482 16172 raft_consensus.cc:686] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e [term 1 LEADER]: Becoming Leader. State: Replica: e7db86a5a92747a7a3ed7090af751e3e, State: Running, Role: LEADER
I20240502 14:13:46.162961 16172 consensus_queue.cc:237] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7db86a5a92747a7a3ed7090af751e3e" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 46165 } }
I20240502 14:13:46.163590 16169 ts_tablet_manager.cc:1398] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20240502 14:13:46.165300 16168 tablet_bootstrap.cc:492] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d: No bootstrap required, opened a new log
I20240502 14:13:46.165474 16168 ts_tablet_manager.cc:1367] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.007s
I20240502 14:13:46.166401 16168 raft_consensus.cc:348] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474981443c8845e59a9ab2348bf59b6d" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 40029 } }
I20240502 14:13:46.166585 16168 raft_consensus.cc:374] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:46.166731 16168 raft_consensus.cc:729] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474981443c8845e59a9ab2348bf59b6d, State: Initialized, Role: FOLLOWER
I20240502 14:13:46.166949 16168 consensus_queue.cc:260] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474981443c8845e59a9ab2348bf59b6d" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 40029 } }
I20240502 14:13:46.167073 16171 tablet_bootstrap.cc:492] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b: No bootstrap required, opened a new log
I20240502 14:13:46.167146 16168 raft_consensus.cc:388] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:46.167241 16168 raft_consensus.cc:482] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:46.167317 16171 ts_tablet_manager.cc:1367] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240502 14:13:46.167363 16168 raft_consensus.cc:3037] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:46.168440 16171 raft_consensus.cc:348] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a8d9bf10df4cc0b22899c97b261a3b" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 44019 } }
I20240502 14:13:46.168685 16171 raft_consensus.cc:374] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:46.168812 16171 raft_consensus.cc:729] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5a8d9bf10df4cc0b22899c97b261a3b, State: Initialized, Role: FOLLOWER
I20240502 14:13:46.169095 16171 consensus_queue.cc:260] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a8d9bf10df4cc0b22899c97b261a3b" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 44019 } }
I20240502 14:13:46.169364 16171 raft_consensus.cc:388] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:46.169510 16171 raft_consensus.cc:482] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:46.169756 16171 raft_consensus.cc:3037] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:46.169744 16168 raft_consensus.cc:504] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474981443c8845e59a9ab2348bf59b6d" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 40029 } }
I20240502 14:13:46.170120 16168 leader_election.cc:302] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 474981443c8845e59a9ab2348bf59b6d; no voters:
I20240502 14:13:46.172765 16171 raft_consensus.cc:504] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a8d9bf10df4cc0b22899c97b261a3b" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 44019 } }
I20240502 14:13:46.173036 16171 leader_election.cc:302] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5a8d9bf10df4cc0b22899c97b261a3b; no voters:
I20240502 14:13:46.173539 16171 leader_election.cc:288] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:46.173758 16180 raft_consensus.cc:2781] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:46.180032 16171 ts_tablet_manager.cc:1398] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b: Time spent starting tablet: real 0.012s user 0.003s sys 0.009s
I20240502 14:13:46.180548 16171 tablet_bootstrap.cc:492] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b: Bootstrap starting.
I20240502 14:13:46.182035 16171 tablet_bootstrap.cc:654] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:46.182807 16179 raft_consensus.cc:2781] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:46.183600 16180 raft_consensus.cc:686] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b [term 1 LEADER]: Becoming Leader. State: Replica: e5a8d9bf10df4cc0b22899c97b261a3b, State: Running, Role: LEADER
I20240502 14:13:46.183993 16168 leader_election.cc:288] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:46.184015 16180 consensus_queue.cc:237] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a8d9bf10df4cc0b22899c97b261a3b" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 44019 } }
I20240502 14:13:46.184788 16171 tablet_bootstrap.cc:492] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b: No bootstrap required, opened a new log
I20240502 14:13:46.185048 16171 ts_tablet_manager.cc:1367] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240502 14:13:46.186206 16171 raft_consensus.cc:348] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a8d9bf10df4cc0b22899c97b261a3b" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 44019 } }
I20240502 14:13:46.186539 16171 raft_consensus.cc:374] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:46.186708 16171 raft_consensus.cc:729] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5a8d9bf10df4cc0b22899c97b261a3b, State: Initialized, Role: FOLLOWER
I20240502 14:13:46.187167 16171 consensus_queue.cc:260] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a8d9bf10df4cc0b22899c97b261a3b" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 44019 } }
I20240502 14:13:46.187570 16171 raft_consensus.cc:388] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:46.189718 16171 raft_consensus.cc:482] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:46.189885 16171 raft_consensus.cc:3037] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:46.193238 16171 raft_consensus.cc:504] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a8d9bf10df4cc0b22899c97b261a3b" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 44019 } }
I20240502 14:13:46.193792 16168 ts_tablet_manager.cc:1398] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d: Time spent starting tablet: real 0.028s user 0.006s sys 0.017s
I20240502 14:13:46.193636 16179 raft_consensus.cc:686] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d [term 1 LEADER]: Becoming Leader. State: Replica: 474981443c8845e59a9ab2348bf59b6d, State: Running, Role: LEADER
I20240502 14:13:46.194175 16179 consensus_queue.cc:237] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474981443c8845e59a9ab2348bf59b6d" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 40029 } }
I20240502 14:13:46.194828 16171 leader_election.cc:302] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5a8d9bf10df4cc0b22899c97b261a3b; no voters:
I20240502 14:13:46.194525 15687 catalog_manager.cc:5484] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b reported cstate change: term changed from 0 to 1, leader changed from <none> to e5a8d9bf10df4cc0b22899c97b261a3b (127.15.74.65). New cstate: current_term: 1 leader_uuid: "e5a8d9bf10df4cc0b22899c97b261a3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a8d9bf10df4cc0b22899c97b261a3b" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 44019 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:46.195183 16171 leader_election.cc:288] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:46.196271 16180 raft_consensus.cc:2781] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:46.197064 16180 raft_consensus.cc:686] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b [term 1 LEADER]: Becoming Leader. State: Replica: e5a8d9bf10df4cc0b22899c97b261a3b, State: Running, Role: LEADER
I20240502 14:13:46.197479 16171 ts_tablet_manager.cc:1398] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s
I20240502 14:13:46.200534 15686 catalog_manager.cc:5484] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e reported cstate change: term changed from 0 to 1, leader changed from <none> to e7db86a5a92747a7a3ed7090af751e3e (127.15.74.67). New cstate: current_term: 1 leader_uuid: "e7db86a5a92747a7a3ed7090af751e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7db86a5a92747a7a3ed7090af751e3e" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 46165 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:46.204733 16180 consensus_queue.cc:237] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a8d9bf10df4cc0b22899c97b261a3b" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 44019 } }
I20240502 14:13:46.211023 15686 catalog_manager.cc:5484] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 474981443c8845e59a9ab2348bf59b6d (127.15.74.66). New cstate: current_term: 1 leader_uuid: "474981443c8845e59a9ab2348bf59b6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474981443c8845e59a9ab2348bf59b6d" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 40029 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:46.212142 15687 catalog_manager.cc:5484] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b reported cstate change: term changed from 0 to 1, leader changed from <none> to e5a8d9bf10df4cc0b22899c97b261a3b (127.15.74.65). New cstate: current_term: 1 leader_uuid: "e5a8d9bf10df4cc0b22899c97b261a3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a8d9bf10df4cc0b22899c97b261a3b" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 44019 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:46.233470 15657 tablet_server.cc:178] TabletServer@127.15.74.65:0 shutting down...
I20240502 14:13:46.245496 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:13:46.245952 15657 tablet_replica.cc:330] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b: stopping tablet replica
I20240502 14:13:46.246376 15657 raft_consensus.cc:2229] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:13:46.246661 15657 raft_consensus.cc:2258] T bca696984f824a9a8f186139150b31ff P e5a8d9bf10df4cc0b22899c97b261a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:46.248220 15657 tablet_replica.cc:330] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b: stopping tablet replica
I20240502 14:13:46.248385 15657 raft_consensus.cc:2229] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:13:46.248534 15657 raft_consensus.cc:2258] T 8a232292622e46318f8846708469e4f6 P e5a8d9bf10df4cc0b22899c97b261a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:46.264166 15657 tablet_server.cc:194] TabletServer@127.15.74.65:0 shutdown complete.
I20240502 14:13:46.269289 15657 tablet_server.cc:178] TabletServer@127.15.74.66:0 shutting down...
I20240502 14:13:46.277657 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:13:46.278013 15657 tablet_replica.cc:330] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d: stopping tablet replica
I20240502 14:13:46.278185 15657 raft_consensus.cc:2229] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:13:46.278362 15657 raft_consensus.cc:2258] T 48368d9e05984de08f1f4f29d2e82986 P 474981443c8845e59a9ab2348bf59b6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:46.292568 15657 tablet_server.cc:194] TabletServer@127.15.74.66:0 shutdown complete.
I20240502 14:13:46.295543 15657 tablet_server.cc:178] TabletServer@127.15.74.67:0 shutting down...
I20240502 14:13:46.304046 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:13:46.304508 15657 tablet_replica.cc:330] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e: stopping tablet replica
I20240502 14:13:46.304718 15657 raft_consensus.cc:2229] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:13:46.304903 15657 raft_consensus.cc:2258] T 6686fa37c473414498b8405cfa421794 P e7db86a5a92747a7a3ed7090af751e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:46.309121 15657 tablet_server.cc:194] TabletServer@127.15.74.67:0 shutdown complete.
I20240502 14:13:46.311892 15657 master.cc:537] Master@127.15.74.126:40475 shutting down...
I20240502 14:13:46.319588 15657 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:13:46.319995 15657 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:46.320160 15657 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3347f91e1d9143c38b128507a98008c5: stopping tablet replica
I20240502 14:13:46.335489 15657 master.cc:552] Master@127.15.74.126:40475 shutdown complete.
I20240502 14:13:46.340884 15657 master.cc:537] Master@127.15.74.125:33081 shutting down...
I20240502 14:13:46.348125 15657 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:46.348353 15657 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:46.348465 15657 tablet_replica.cc:330] T 00000000000000000000000000000000 P 71adf07ec4984de784bb91f1855e03ca: stopping tablet replica
I20240502 14:13:46.362366 15657 master.cc:552] Master@127.15.74.125:33081 shutdown complete.
I20240502 14:13:46.366508 15657 master.cc:537] Master@127.15.74.124:38935 shutting down...
I20240502 14:13:46.374272 15657 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:46.374516 15657 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:46.374650 15657 tablet_replica.cc:330] T 00000000000000000000000000000000 P 562dbd4b638a4e318bf48f05660e401e: stopping tablet replica
I20240502 14:13:46.388396 15657 master.cc:552] Master@127.15.74.124:38935 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (3092 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240502 14:13:46.401448 15657 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.74.126:36099
I20240502 14:13:46.401949 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:46.405009 16191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:46.405737 16192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:46.406464 16194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:46.407167 15657 server_base.cc:1034] running on GCE node
I20240502 14:13:46.407490 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:46.407577 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:46.407639 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659226407618 us; error 0 us; skew 500 ppm
I20240502 14:13:46.407807 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:46.409005 15657 webserver.cc:460] Webserver started at http://127.15.74.126:39199/ using document root <none> and password file <none>
I20240502 14:13:46.409230 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:46.409308 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:46.409490 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:46.410130 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/master-0-root/instance:
uuid: "6894cdd189fe417188e31a90198edbe6"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-wbm6"
I20240502 14:13:46.412729 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:13:46.414355 16199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:46.414737 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:13:46.414844 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/master-0-root
uuid: "6894cdd189fe417188e31a90198edbe6"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-wbm6"
I20240502 14:13:46.414940 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:46.429476 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:46.430050 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:46.444175 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.126:36099
I20240502 14:13:46.444204 16250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.126:36099 every 8 connection(s)
I20240502 14:13:46.445525 16251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:46.449357 16251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6: Bootstrap starting.
I20240502 14:13:46.450596 16251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:46.452361 16251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6: No bootstrap required, opened a new log
I20240502 14:13:46.452981 16251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6894cdd189fe417188e31a90198edbe6" member_type: VOTER }
I20240502 14:13:46.453121 16251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:46.453238 16251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6894cdd189fe417188e31a90198edbe6, State: Initialized, Role: FOLLOWER
I20240502 14:13:46.453430 16251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6894cdd189fe417188e31a90198edbe6" member_type: VOTER }
I20240502 14:13:46.453603 16251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:46.453693 16251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:46.453823 16251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:46.455709 16251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6894cdd189fe417188e31a90198edbe6" member_type: VOTER }
I20240502 14:13:46.455917 16251 leader_election.cc:302] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6894cdd189fe417188e31a90198edbe6; no voters:
I20240502 14:13:46.456315 16251 leader_election.cc:288] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:46.456431 16254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:46.456912 16254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [term 1 LEADER]: Becoming Leader. State: Replica: 6894cdd189fe417188e31a90198edbe6, State: Running, Role: LEADER
I20240502 14:13:46.457193 16251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:13:46.457171 16254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6894cdd189fe417188e31a90198edbe6" member_type: VOTER }
I20240502 14:13:46.458058 16256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6894cdd189fe417188e31a90198edbe6. Latest consensus state: current_term: 1 leader_uuid: "6894cdd189fe417188e31a90198edbe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6894cdd189fe417188e31a90198edbe6" member_type: VOTER } }
I20240502 14:13:46.458029 16255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6894cdd189fe417188e31a90198edbe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6894cdd189fe417188e31a90198edbe6" member_type: VOTER } }
I20240502 14:13:46.458266 16256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:46.458379 16255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:46.459776 16259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:13:46.462002 16259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:13:46.462493 15657 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:13:46.464941 16259 catalog_manager.cc:1347] Generated new cluster ID: 05632a4d82304741a49309b7af9c74e8
I20240502 14:13:46.465024 16259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:13:46.483042 16259 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:13:46.483853 16259 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:13:46.499394 16259 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6: Generated new TSK 0
I20240502 14:13:46.499702 16259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:13:46.527740 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:46.531580 16272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:46.532119 16273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:46.532763 16275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:46.533702 15657 server_base.cc:1034] running on GCE node
I20240502 14:13:46.533977 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:46.534042 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:46.534116 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659226534079 us; error 0 us; skew 500 ppm
I20240502 14:13:46.534267 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:46.535614 15657 webserver.cc:460] Webserver started at http://127.15.74.65:46717/ using document root <none> and password file <none>
I20240502 14:13:46.535881 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:46.535984 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:46.536131 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:46.536854 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-0-root/instance:
uuid: "fe33bde6df0245cb84db702ff5a13e11"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-wbm6"
I20240502 14:13:46.539456 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 14:13:46.541164 16280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:46.541538 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240502 14:13:46.541651 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-0-root
uuid: "fe33bde6df0245cb84db702ff5a13e11"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-wbm6"
I20240502 14:13:46.541786 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:46.549100 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:46.549707 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:46.550299 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:46.551187 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:46.551259 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:46.551376 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:46.551465 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:46.566924 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.65:34993
I20240502 14:13:46.566956 16342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.65:34993 every 8 connection(s)
I20240502 14:13:46.568737 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:46.572518 16347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:46.573403 16348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:46.574155 16350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:46.574968 15657 server_base.cc:1034] running on GCE node
I20240502 14:13:46.575266 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:46.575367 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:46.575456 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659226575413 us; error 0 us; skew 500 ppm
I20240502 14:13:46.575652 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:46.576478 16343 heartbeater.cc:347] Connected to a master server at 127.15.74.126:36099
I20240502 14:13:46.576611 16343 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:46.577014 16343 heartbeater.cc:511] Master 127.15.74.126:36099 requested a full tablet report, sending...
I20240502 14:13:46.577139 15657 webserver.cc:460] Webserver started at http://127.15.74.66:40097/ using document root <none> and password file <none>
I20240502 14:13:46.577390 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:46.577484 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:46.577648 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:46.577986 16216 ts_manager.cc:194] Registered new tserver with Master: fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65:34993)
I20240502 14:13:46.578402 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-1-root/instance:
uuid: "644249615cac43b9af8f167605f3437c"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-wbm6"
I20240502 14:13:46.579114 16216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38196
I20240502 14:13:46.581040 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20240502 14:13:46.582612 16355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:46.583003 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 14:13:46.583122 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-1-root
uuid: "644249615cac43b9af8f167605f3437c"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-wbm6"
I20240502 14:13:46.583222 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:46.595830 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:46.596355 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:46.596870 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:46.597642 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:46.597712 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:46.597774 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:46.597822 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:46.612367 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.66:44425
I20240502 14:13:46.612407 16417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.66:44425 every 8 connection(s)
I20240502 14:13:46.614280 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:46.617699 16421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:46.619894 16422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:46.620764 16424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:46.621369 15657 server_base.cc:1034] running on GCE node
I20240502 14:13:46.621647 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:46.621730 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:46.621812 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659226621772 us; error 0 us; skew 500 ppm
I20240502 14:13:46.621981 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:46.622241 16418 heartbeater.cc:347] Connected to a master server at 127.15.74.126:36099
I20240502 14:13:46.622387 16418 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:46.622818 16418 heartbeater.cc:511] Master 127.15.74.126:36099 requested a full tablet report, sending...
I20240502 14:13:46.623412 15657 webserver.cc:460] Webserver started at http://127.15.74.67:46561/ using document root <none> and password file <none>
I20240502 14:13:46.623746 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:46.623873 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:46.623910 16216 ts_manager.cc:194] Registered new tserver with Master: 644249615cac43b9af8f167605f3437c (127.15.74.66:44425)
I20240502 14:13:46.624040 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:46.624917 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-2-root/instance:
uuid: "ef0b0185fda54117bb2b1a2f7672f386"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-wbm6"
I20240502 14:13:46.625005 16216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38202
I20240502 14:13:46.627673 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20240502 14:13:46.629207 16429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:46.629603 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 14:13:46.629703 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-2-root
uuid: "ef0b0185fda54117bb2b1a2f7672f386"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-wbm6"
I20240502 14:13:46.629841 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:46.646551 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:46.647159 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:46.647780 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:46.648635 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:46.648710 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:46.648824 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:46.648880 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:46.663039 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.67:43455
I20240502 14:13:46.663062 16491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.67:43455 every 8 connection(s)
I20240502 14:13:46.669858 16492 heartbeater.cc:347] Connected to a master server at 127.15.74.126:36099
I20240502 14:13:46.669994 16492 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:46.670397 16492 heartbeater.cc:511] Master 127.15.74.126:36099 requested a full tablet report, sending...
I20240502 14:13:46.671211 16216 ts_manager.cc:194] Registered new tserver with Master: ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67:43455)
I20240502 14:13:46.671828 15657 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007906739s
I20240502 14:13:46.672091 16216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38212
I20240502 14:13:47.580396 16343 heartbeater.cc:503] Master 127.15.74.126:36099 was elected leader, sending a full tablet report...
I20240502 14:13:47.626591 16418 heartbeater.cc:503] Master 127.15.74.126:36099 was elected leader, sending a full tablet report...
I20240502 14:13:47.673650 16492 heartbeater.cc:503] Master 127.15.74.126:36099 was elected leader, sending a full tablet report...
I20240502 14:13:47.699972 15657 test_util.cc:268] Using random seed: 908954342
I20240502 14:13:47.714012 16216 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38216:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:13:47.714737 16216 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:13:47.747251 16308 tablet_service.cc:1467] Processing CreateTablet for tablet eb57739bd6504281a0ab8060a69745eb (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 14:13:47.747792 16303 tablet_service.cc:1467] Processing CreateTablet for tablet 8d908982bfbc402c8892040b69d2a81c (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 14:13:47.747918 16308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb57739bd6504281a0ab8060a69745eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.748030 16307 tablet_service.cc:1467] Processing CreateTablet for tablet c42b20951bc7478d8b01231c46995aa9 (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 14:13:47.748250 16303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d908982bfbc402c8892040b69d2a81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.748476 16307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42b20951bc7478d8b01231c46995aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.748669 16306 tablet_service.cc:1467] Processing CreateTablet for tablet 05f15bd32e5a45f6bbe297bb4daaaf8a (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 14:13:47.749133 16306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f15bd32e5a45f6bbe297bb4daaaf8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.751861 16305 tablet_service.cc:1467] Processing CreateTablet for tablet 80179381859d4afcb39f7c378313c9a1 (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 14:13:47.752321 16305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80179381859d4afcb39f7c378313c9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.755029 16301 tablet_service.cc:1467] Processing CreateTablet for tablet bfc19f54c7e4409a8a0ecde5be05cb48 (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 14:13:47.756047 16301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc19f54c7e4409a8a0ecde5be05cb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.755035 16512 tablet_bootstrap.cc:492] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11: Bootstrap starting.
I20240502 14:13:47.757737 16304 tablet_service.cc:1467] Processing CreateTablet for tablet 8f5963f36815436bba21124bd3a7e9d3 (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 14:13:47.758791 16382 tablet_service.cc:1467] Processing CreateTablet for tablet c42b20951bc7478d8b01231c46995aa9 (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 14:13:47.760243 16304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f5963f36815436bba21124bd3a7e9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.759677 16381 tablet_service.cc:1467] Processing CreateTablet for tablet 05f15bd32e5a45f6bbe297bb4daaaf8a (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 14:13:47.760845 16382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42b20951bc7478d8b01231c46995aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.760247 16302 tablet_service.cc:1467] Processing CreateTablet for tablet 920ffab6592f45c795bbd8789de768a4 (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 14:13:47.760788 16383 tablet_service.cc:1467] Processing CreateTablet for tablet eb57739bd6504281a0ab8060a69745eb (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 14:13:47.762202 16378 tablet_service.cc:1467] Processing CreateTablet for tablet 8d908982bfbc402c8892040b69d2a81c (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 14:13:47.762452 16381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f15bd32e5a45f6bbe297bb4daaaf8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.764081 16302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920ffab6592f45c795bbd8789de768a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.764133 16379 tablet_service.cc:1467] Processing CreateTablet for tablet 8f5963f36815436bba21124bd3a7e9d3 (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 14:13:47.766076 16379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f5963f36815436bba21124bd3a7e9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.765951 16376 tablet_service.cc:1467] Processing CreateTablet for tablet bfc19f54c7e4409a8a0ecde5be05cb48 (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 14:13:47.766814 16380 tablet_service.cc:1467] Processing CreateTablet for tablet 80179381859d4afcb39f7c378313c9a1 (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 14:13:47.767577 16377 tablet_service.cc:1467] Processing CreateTablet for tablet 920ffab6592f45c795bbd8789de768a4 (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 14:13:47.768235 16378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d908982bfbc402c8892040b69d2a81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.768862 16383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb57739bd6504281a0ab8060a69745eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.769698 16376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc19f54c7e4409a8a0ecde5be05cb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.769881 16377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920ffab6592f45c795bbd8789de768a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.770503 16380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80179381859d4afcb39f7c378313c9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.788906 16512 tablet_bootstrap.cc:654] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.795936 16457 tablet_service.cc:1467] Processing CreateTablet for tablet eb57739bd6504281a0ab8060a69745eb (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 14:13:47.799940 16455 tablet_service.cc:1467] Processing CreateTablet for tablet 05f15bd32e5a45f6bbe297bb4daaaf8a (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 14:13:47.800969 16454 tablet_service.cc:1467] Processing CreateTablet for tablet 80179381859d4afcb39f7c378313c9a1 (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 14:13:47.801682 16453 tablet_service.cc:1467] Processing CreateTablet for tablet 8f5963f36815436bba21124bd3a7e9d3 (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 14:13:47.802264 16456 tablet_service.cc:1467] Processing CreateTablet for tablet c42b20951bc7478d8b01231c46995aa9 (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 14:13:47.805182 16452 tablet_service.cc:1467] Processing CreateTablet for tablet 8d908982bfbc402c8892040b69d2a81c (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 14:13:47.805449 16457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb57739bd6504281a0ab8060a69745eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.807096 16450 tablet_service.cc:1467] Processing CreateTablet for tablet bfc19f54c7e4409a8a0ecde5be05cb48 (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 14:13:47.807500 16451 tablet_service.cc:1467] Processing CreateTablet for tablet 920ffab6592f45c795bbd8789de768a4 (DEFAULT_TABLE table=test-workload [id=ae950d2d72e943faa0e44ed548926d7b]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 14:13:47.808043 16455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f15bd32e5a45f6bbe297bb4daaaf8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.809209 16456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42b20951bc7478d8b01231c46995aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.809491 16454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80179381859d4afcb39f7c378313c9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.810039 16451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920ffab6592f45c795bbd8789de768a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.810536 16450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc19f54c7e4409a8a0ecde5be05cb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.811517 16453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f5963f36815436bba21124bd3a7e9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.815126 16452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d908982bfbc402c8892040b69d2a81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.815407 16513 tablet_bootstrap.cc:492] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c: Bootstrap starting.
I20240502 14:13:47.832777 16513 tablet_bootstrap.cc:654] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.839455 16513 tablet_bootstrap.cc:492] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c: No bootstrap required, opened a new log
I20240502 14:13:47.839661 16513 ts_tablet_manager.cc:1367] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.008s
I20240502 14:13:47.844873 16513 raft_consensus.cc:348] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.845113 16513 raft_consensus.cc:374] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.847471 16513 raft_consensus.cc:729] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644249615cac43b9af8f167605f3437c, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.848366 16512 tablet_bootstrap.cc:492] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11: No bootstrap required, opened a new log
I20240502 14:13:47.848524 16512 ts_tablet_manager.cc:1367] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11: Time spent bootstrapping tablet: real 0.094s user 0.008s sys 0.017s
I20240502 14:13:47.849185 16512 raft_consensus.cc:348] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.849372 16512 raft_consensus.cc:374] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.849495 16512 raft_consensus.cc:729] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe33bde6df0245cb84db702ff5a13e11, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.849747 16512 consensus_queue.cc:260] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.850160 16513 consensus_queue.cc:260] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.853369 16512 ts_tablet_manager.cc:1398] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:13:47.854274 16512 tablet_bootstrap.cc:492] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11: Bootstrap starting.
I20240502 14:13:47.857539 16512 tablet_bootstrap.cc:654] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.858783 16515 tablet_bootstrap.cc:492] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386: Bootstrap starting.
I20240502 14:13:47.858789 16513 ts_tablet_manager.cc:1398] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c: Time spent starting tablet: real 0.019s user 0.000s sys 0.005s
I20240502 14:13:47.860177 16513 tablet_bootstrap.cc:492] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c: Bootstrap starting.
I20240502 14:13:47.862313 16515 tablet_bootstrap.cc:654] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.862334 16513 tablet_bootstrap.cc:654] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.862850 16512 tablet_bootstrap.cc:492] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11: No bootstrap required, opened a new log
I20240502 14:13:47.863073 16512 ts_tablet_manager.cc:1367] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20240502 14:13:47.863848 16512 raft_consensus.cc:348] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.864023 16512 raft_consensus.cc:374] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.864084 16512 raft_consensus.cc:729] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe33bde6df0245cb84db702ff5a13e11, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.864333 16512 consensus_queue.cc:260] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.864866 16512 ts_tablet_manager.cc:1398] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:13:47.865161 16512 tablet_bootstrap.cc:492] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11: Bootstrap starting.
I20240502 14:13:47.866855 16512 tablet_bootstrap.cc:654] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.869181 16515 tablet_bootstrap.cc:492] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386: No bootstrap required, opened a new log
I20240502 14:13:47.869369 16515 ts_tablet_manager.cc:1367] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240502 14:13:47.869752 16513 tablet_bootstrap.cc:492] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c: No bootstrap required, opened a new log
I20240502 14:13:47.869907 16513 ts_tablet_manager.cc:1367] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20240502 14:13:47.870014 16512 tablet_bootstrap.cc:492] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11: No bootstrap required, opened a new log
I20240502 14:13:47.870177 16512 ts_tablet_manager.cc:1367] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 14:13:47.870245 16515 raft_consensus.cc:348] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.870474 16515 raft_consensus.cc:374] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.870591 16515 raft_consensus.cc:729] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0b0185fda54117bb2b1a2f7672f386, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.870678 16513 raft_consensus.cc:348] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.870924 16513 raft_consensus.cc:374] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.870851 16515 consensus_queue.cc:260] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.871037 16513 raft_consensus.cc:729] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644249615cac43b9af8f167605f3437c, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.870899 16512 raft_consensus.cc:348] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.871162 16512 raft_consensus.cc:374] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.871273 16512 raft_consensus.cc:729] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe33bde6df0245cb84db702ff5a13e11, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.871368 16513 consensus_queue.cc:260] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.871549 16512 consensus_queue.cc:260] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.872054 16512 ts_tablet_manager.cc:1398] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:13:47.872330 16512 tablet_bootstrap.cc:492] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11: Bootstrap starting.
I20240502 14:13:47.873811 16513 ts_tablet_manager.cc:1398] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:13:47.874045 16512 tablet_bootstrap.cc:654] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.874187 16513 tablet_bootstrap.cc:492] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c: Bootstrap starting.
I20240502 14:13:47.876222 16513 tablet_bootstrap.cc:654] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.880342 16515 ts_tablet_manager.cc:1398] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386: Time spent starting tablet: real 0.011s user 0.003s sys 0.007s
I20240502 14:13:47.881636 16515 tablet_bootstrap.cc:492] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386: Bootstrap starting.
I20240502 14:13:47.883761 16515 tablet_bootstrap.cc:654] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.885979 16512 tablet_bootstrap.cc:492] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11: No bootstrap required, opened a new log
I20240502 14:13:47.886166 16512 ts_tablet_manager.cc:1367] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.012s
I20240502 14:13:47.886569 16513 tablet_bootstrap.cc:492] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c: No bootstrap required, opened a new log
I20240502 14:13:47.886677 16513 ts_tablet_manager.cc:1367] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20240502 14:13:47.886904 16512 raft_consensus.cc:348] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.887146 16512 raft_consensus.cc:374] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.887238 16512 raft_consensus.cc:729] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe33bde6df0245cb84db702ff5a13e11, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.887198 16513 raft_consensus.cc:348] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.887290 16515 tablet_bootstrap.cc:492] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386: No bootstrap required, opened a new log
I20240502 14:13:47.887341 16513 raft_consensus.cc:374] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.887533 16513 raft_consensus.cc:729] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644249615cac43b9af8f167605f3437c, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.887661 16515 ts_tablet_manager.cc:1367] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:13:47.887578 16512 consensus_queue.cc:260] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.887738 16513 consensus_queue.cc:260] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.888106 16512 ts_tablet_manager.cc:1398] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:13:47.888459 16512 tablet_bootstrap.cc:492] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11: Bootstrap starting.
I20240502 14:13:47.888659 16515 raft_consensus.cc:348] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.888950 16515 raft_consensus.cc:374] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.888984 16513 ts_tablet_manager.cc:1398] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:13:47.889145 16515 raft_consensus.cc:729] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0b0185fda54117bb2b1a2f7672f386, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.889425 16513 tablet_bootstrap.cc:492] T 920ffab6592f45c795bbd8789de768a4 P 644249615cac43b9af8f167605f3437c: Bootstrap starting.
I20240502 14:13:47.889582 16515 consensus_queue.cc:260] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.890237 16512 tablet_bootstrap.cc:654] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.890237 16515 ts_tablet_manager.cc:1398] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:13:47.890978 16513 tablet_bootstrap.cc:654] T 920ffab6592f45c795bbd8789de768a4 P 644249615cac43b9af8f167605f3437c: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.891458 16515 tablet_bootstrap.cc:492] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386: Bootstrap starting.
I20240502 14:13:47.893532 16515 tablet_bootstrap.cc:654] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.897517 16518 raft_consensus.cc:482] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:47.897670 16518 raft_consensus.cc:504] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.898761 16518 leader_election.cc:288] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65:34993), ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67:43455)
I20240502 14:13:47.901239 16512 tablet_bootstrap.cc:492] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11: No bootstrap required, opened a new log
I20240502 14:13:47.901418 16512 ts_tablet_manager.cc:1367] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20240502 14:13:47.902029 16515 tablet_bootstrap.cc:492] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386: No bootstrap required, opened a new log
I20240502 14:13:47.902236 16515 ts_tablet_manager.cc:1367] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20240502 14:13:47.902237 16512 raft_consensus.cc:348] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.902495 16512 raft_consensus.cc:374] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.902654 16512 raft_consensus.cc:729] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe33bde6df0245cb84db702ff5a13e11, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.902889 16512 consensus_queue.cc:260] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.903309 16515 raft_consensus.cc:348] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.903452 16512 ts_tablet_manager.cc:1398] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:13:47.903616 16515 raft_consensus.cc:374] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.903781 16515 raft_consensus.cc:729] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0b0185fda54117bb2b1a2f7672f386, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.903827 16512 tablet_bootstrap.cc:492] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11: Bootstrap starting.
I20240502 14:13:47.904157 16515 consensus_queue.cc:260] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.905448 16512 tablet_bootstrap.cc:654] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.905675 16515 ts_tablet_manager.cc:1398] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:13:47.906015 16515 tablet_bootstrap.cc:492] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386: Bootstrap starting.
I20240502 14:13:47.908089 16515 tablet_bootstrap.cc:654] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.911808 16517 raft_consensus.cc:482] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:47.911983 16517 raft_consensus.cc:504] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.913236 16517 leader_election.cc:288] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 644249615cac43b9af8f167605f3437c (127.15.74.66:44425), ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67:43455)
I20240502 14:13:47.917510 16515 tablet_bootstrap.cc:492] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386: No bootstrap required, opened a new log
I20240502 14:13:47.917795 16515 ts_tablet_manager.cc:1367] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20240502 14:13:47.918138 16513 tablet_bootstrap.cc:492] T 920ffab6592f45c795bbd8789de768a4 P 644249615cac43b9af8f167605f3437c: No bootstrap required, opened a new log
I20240502 14:13:47.918340 16513 ts_tablet_manager.cc:1367] T 920ffab6592f45c795bbd8789de768a4 P 644249615cac43b9af8f167605f3437c: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.005s
I20240502 14:13:47.918946 16515 raft_consensus.cc:348] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.919306 16515 raft_consensus.cc:374] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.919481 16515 raft_consensus.cc:729] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0b0185fda54117bb2b1a2f7672f386, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.919966 16515 consensus_queue.cc:260] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.921859 16515 ts_tablet_manager.cc:1398] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:13:47.922317 16515 tablet_bootstrap.cc:492] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386: Bootstrap starting.
I20240502 14:13:47.919181 16513 raft_consensus.cc:348] T 920ffab6592f45c795bbd8789de768a4 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.923295 16513 raft_consensus.cc:374] T 920ffab6592f45c795bbd8789de768a4 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.923415 16513 raft_consensus.cc:729] T 920ffab6592f45c795bbd8789de768a4 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644249615cac43b9af8f167605f3437c, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.924235 16513 consensus_queue.cc:260] T 920ffab6592f45c795bbd8789de768a4 P 644249615cac43b9af8f167605f3437c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.924942 16515 tablet_bootstrap.cc:654] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.926805 16513 ts_tablet_manager.cc:1398] T 920ffab6592f45c795bbd8789de768a4 P 644249615cac43b9af8f167605f3437c: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240502 14:13:47.928809 16513 tablet_bootstrap.cc:492] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c: Bootstrap starting.
I20240502 14:13:47.930541 16513 tablet_bootstrap.cc:654] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.931103 16523 raft_consensus.cc:482] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:47.931322 16523 raft_consensus.cc:504] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.932744 16523 leader_election.cc:288] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65:34993), 644249615cac43b9af8f167605f3437c (127.15.74.66:44425)
I20240502 14:13:47.934348 16523 raft_consensus.cc:482] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:47.934520 16523 raft_consensus.cc:504] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.935307 16523 leader_election.cc:288] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65:34993), 644249615cac43b9af8f167605f3437c (127.15.74.66:44425)
I20240502 14:13:47.944538 16523 raft_consensus.cc:482] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:47.944815 16523 raft_consensus.cc:504] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.945705 16523 leader_election.cc:288] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65:34993), 644249615cac43b9af8f167605f3437c (127.15.74.66:44425)
I20240502 14:13:47.947928 16467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42b20951bc7478d8b01231c46995aa9" candidate_uuid: "644249615cac43b9af8f167605f3437c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0b0185fda54117bb2b1a2f7672f386" is_pre_election: true
W20240502 14:13:47.949216 16359 leader_election.cc:341] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67:43455): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:13:47.952170 16515 tablet_bootstrap.cc:492] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386: No bootstrap required, opened a new log
I20240502 14:13:47.952447 16515 ts_tablet_manager.cc:1367] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.005s
I20240502 14:13:47.953467 16515 raft_consensus.cc:348] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.953680 16515 raft_consensus.cc:374] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.953795 16515 raft_consensus.cc:729] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0b0185fda54117bb2b1a2f7672f386, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.954061 16515 consensus_queue.cc:260] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.952209 16513 tablet_bootstrap.cc:492] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c: No bootstrap required, opened a new log
I20240502 14:13:47.954424 16513 ts_tablet_manager.cc:1367] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20240502 14:13:47.954576 16512 tablet_bootstrap.cc:492] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11: No bootstrap required, opened a new log
I20240502 14:13:47.954749 16512 ts_tablet_manager.cc:1367] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11: Time spent bootstrapping tablet: real 0.051s user 0.015s sys 0.030s
I20240502 14:13:47.955291 16513 raft_consensus.cc:348] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.955372 16512 raft_consensus.cc:348] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } }
I20240502 14:13:47.955551 16513 raft_consensus.cc:374] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.955682 16513 raft_consensus.cc:729] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644249615cac43b9af8f167605f3437c, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.955969 16513 consensus_queue.cc:260] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.955580 16512 raft_consensus.cc:374] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.956521 16513 ts_tablet_manager.cc:1398] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:13:47.956599 16512 raft_consensus.cc:729] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe33bde6df0245cb84db702ff5a13e11, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.956969 16513 tablet_bootstrap.cc:492] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c: Bootstrap starting.
I20240502 14:13:47.956929 16512 consensus_queue.cc:260] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } }
I20240502 14:13:47.957607 16512 ts_tablet_manager.cc:1398] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:13:47.957942 16512 tablet_bootstrap.cc:492] T 920ffab6592f45c795bbd8789de768a4 P fe33bde6df0245cb84db702ff5a13e11: Bootstrap starting.
I20240502 14:13:47.960489 16517 raft_consensus.cc:482] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:47.961946 16517 raft_consensus.cc:504] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.962450 16517 leader_election.cc:288] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 644249615cac43b9af8f167605f3437c (127.15.74.66:44425), ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67:43455)
I20240502 14:13:47.962658 16515 ts_tablet_manager.cc:1398] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:13:47.963215 16513 tablet_bootstrap.cc:654] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.965229 16512 tablet_bootstrap.cc:654] T 920ffab6592f45c795bbd8789de768a4 P fe33bde6df0245cb84db702ff5a13e11: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.965816 16515 tablet_bootstrap.cc:492] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386: Bootstrap starting.
I20240502 14:13:47.972890 16523 raft_consensus.cc:482] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:47.973934 16523 raft_consensus.cc:504] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.975137 16523 leader_election.cc:288] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65:34993), 644249615cac43b9af8f167605f3437c (127.15.74.66:44425)
I20240502 14:13:47.976986 16513 tablet_bootstrap.cc:492] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c: No bootstrap required, opened a new log
I20240502 14:13:47.973659 16318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42b20951bc7478d8b01231c46995aa9" candidate_uuid: "644249615cac43b9af8f167605f3437c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe33bde6df0245cb84db702ff5a13e11" is_pre_election: true
I20240502 14:13:47.979487 16318 raft_consensus.cc:2454] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 644249615cac43b9af8f167605f3437c in term 0.
I20240502 14:13:47.979928 16513 ts_tablet_manager.cc:1367] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.003s
I20240502 14:13:47.978899 16392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d908982bfbc402c8892040b69d2a81c" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c" is_pre_election: true
I20240502 14:13:47.980911 16515 tablet_bootstrap.cc:654] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:47.981801 16358 leader_election.cc:302] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 644249615cac43b9af8f167605f3437c, fe33bde6df0245cb84db702ff5a13e11; no voters: ef0b0185fda54117bb2b1a2f7672f386
I20240502 14:13:47.982120 16518 raft_consensus.cc:2781] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:13:47.982249 16518 raft_consensus.cc:482] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:47.982887 16518 raft_consensus.cc:3037] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:47.980429 16393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920ffab6592f45c795bbd8789de768a4" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c" is_pre_election: true
I20240502 14:13:47.983618 16393 raft_consensus.cc:2454] T 920ffab6592f45c795bbd8789de768a4 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0b0185fda54117bb2b1a2f7672f386 in term 0.
I20240502 14:13:47.983414 16512 tablet_bootstrap.cc:492] T 920ffab6592f45c795bbd8789de768a4 P fe33bde6df0245cb84db702ff5a13e11: No bootstrap required, opened a new log
I20240502 14:13:47.981305 16391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc19f54c7e4409a8a0ecde5be05cb48" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c" is_pre_election: true
I20240502 14:13:47.984359 16512 ts_tablet_manager.cc:1367] T 920ffab6592f45c795bbd8789de768a4 P fe33bde6df0245cb84db702ff5a13e11: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.006s
I20240502 14:13:47.985344 16431 leader_election.cc:302] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 644249615cac43b9af8f167605f3437c, ef0b0185fda54117bb2b1a2f7672f386; no voters:
I20240502 14:13:47.983539 16390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f15bd32e5a45f6bbe297bb4daaaf8a" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c" is_pre_election: true
I20240502 14:13:47.985622 16390 raft_consensus.cc:2454] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0b0185fda54117bb2b1a2f7672f386 in term 0.
W20240502 14:13:47.985685 16431 leader_election.cc:341] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 644249615cac43b9af8f167605f3437c (127.15.74.66:44425): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:13:47.985802 16513 raft_consensus.cc:348] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.986356 16523 raft_consensus.cc:2781] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:13:47.986771 16431 leader_election.cc:302] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 644249615cac43b9af8f167605f3437c, ef0b0185fda54117bb2b1a2f7672f386; no voters:
I20240502 14:13:47.986766 16523 raft_consensus.cc:482] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:47.987243 16523 raft_consensus.cc:3037] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:47.987634 16513 raft_consensus.cc:374] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:47.988557 16513 raft_consensus.cc:729] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644249615cac43b9af8f167605f3437c, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.997578 16523 raft_consensus.cc:504] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:47.999464 16523 leader_election.cc:288] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 election: Requested vote from peers fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65:34993), 644249615cac43b9af8f167605f3437c (127.15.74.66:44425)
I20240502 14:13:47.999696 16518 raft_consensus.cc:504] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.000384 16518 leader_election.cc:288] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [CANDIDATE]: Term 1 election: Requested vote from peers fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65:34993), ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67:43455)
I20240502 14:13:47.999191 16513 consensus_queue.cc:260] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.001282 16392 raft_consensus.cc:2454] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0b0185fda54117bb2b1a2f7672f386 in term 0.
I20240502 14:13:48.005553 16523 raft_consensus.cc:2781] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:13:48.006789 16523 raft_consensus.cc:482] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:48.006919 16523 raft_consensus.cc:3037] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.007493 16467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42b20951bc7478d8b01231c46995aa9" candidate_uuid: "644249615cac43b9af8f167605f3437c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0b0185fda54117bb2b1a2f7672f386"
I20240502 14:13:48.008513 16431 leader_election.cc:302] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 644249615cac43b9af8f167605f3437c, ef0b0185fda54117bb2b1a2f7672f386; no voters:
I20240502 14:13:48.008695 16391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920ffab6592f45c795bbd8789de768a4" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c"
I20240502 14:13:48.009033 16318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42b20951bc7478d8b01231c46995aa9" candidate_uuid: "644249615cac43b9af8f167605f3437c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe33bde6df0245cb84db702ff5a13e11"
I20240502 14:13:48.009192 16512 raft_consensus.cc:348] T 920ffab6592f45c795bbd8789de768a4 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.009450 16513 ts_tablet_manager.cc:1398] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c: Time spent starting tablet: real 0.026s user 0.000s sys 0.003s
I20240502 14:13:48.009835 16392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc19f54c7e4409a8a0ecde5be05cb48" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c" is_pre_election: true
W20240502 14:13:48.010062 16359 leader_election.cc:341] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67:43455): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:13:48.010668 16390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f15bd32e5a45f6bbe297bb4daaaf8a" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c" is_pre_election: true
I20240502 14:13:48.010838 16318 raft_consensus.cc:3037] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.011059 16512 raft_consensus.cc:374] T 920ffab6592f45c795bbd8789de768a4 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:48.011958 16390 raft_consensus.cc:2454] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe33bde6df0245cb84db702ff5a13e11 in term 0.
I20240502 14:13:48.011314 16466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f15bd32e5a45f6bbe297bb4daaaf8a" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0b0185fda54117bb2b1a2f7672f386" is_pre_election: true
I20240502 14:13:48.013676 16467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc19f54c7e4409a8a0ecde5be05cb48" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0b0185fda54117bb2b1a2f7672f386" is_pre_election: true
I20240502 14:13:48.014220 16391 raft_consensus.cc:3037] T 920ffab6592f45c795bbd8789de768a4 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.014406 16467 raft_consensus.cc:2454] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe33bde6df0245cb84db702ff5a13e11 in term 0.
W20240502 14:13:48.015357 16282 leader_election.cc:341] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 644249615cac43b9af8f167605f3437c (127.15.74.66:44425): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:13:48.015906 16515 tablet_bootstrap.cc:492] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386: No bootstrap required, opened a new log
I20240502 14:13:48.015004 16317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920ffab6592f45c795bbd8789de768a4" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe33bde6df0245cb84db702ff5a13e11" is_pre_election: true
I20240502 14:13:48.016644 16544 raft_consensus.cc:2781] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:13:48.017365 16282 leader_election.cc:302] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 644249615cac43b9af8f167605f3437c, fe33bde6df0245cb84db702ff5a13e11; no voters:
I20240502 14:13:48.017696 16513 tablet_bootstrap.cc:492] T bfc19f54c7e4409a8a0ecde5be05cb48 P 644249615cac43b9af8f167605f3437c: Bootstrap starting.
I20240502 14:13:48.016702 16316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d908982bfbc402c8892040b69d2a81c" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe33bde6df0245cb84db702ff5a13e11" is_pre_election: true
I20240502 14:13:48.017292 16315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc19f54c7e4409a8a0ecde5be05cb48" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe33bde6df0245cb84db702ff5a13e11" is_pre_election: true
I20240502 14:13:48.018122 16316 raft_consensus.cc:2454] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0b0185fda54117bb2b1a2f7672f386 in term 0.
I20240502 14:13:48.018162 16315 raft_consensus.cc:2454] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0b0185fda54117bb2b1a2f7672f386 in term 0.
I20240502 14:13:48.019492 16513 tablet_bootstrap.cc:654] T bfc19f54c7e4409a8a0ecde5be05cb48 P 644249615cac43b9af8f167605f3437c: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:48.016703 16512 raft_consensus.cc:729] T 920ffab6592f45c795bbd8789de768a4 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe33bde6df0245cb84db702ff5a13e11, State: Initialized, Role: FOLLOWER
I20240502 14:13:48.019827 16512 consensus_queue.cc:260] T 920ffab6592f45c795bbd8789de768a4 P fe33bde6df0245cb84db702ff5a13e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.020119 16317 raft_consensus.cc:2454] T 920ffab6592f45c795bbd8789de768a4 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0b0185fda54117bb2b1a2f7672f386 in term 0.
I20240502 14:13:48.020268 16391 raft_consensus.cc:2454] T 920ffab6592f45c795bbd8789de768a4 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0b0185fda54117bb2b1a2f7672f386 in term 1.
I20240502 14:13:48.021230 16544 raft_consensus.cc:482] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:48.021835 16515 ts_tablet_manager.cc:1367] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386: Time spent bootstrapping tablet: real 0.053s user 0.013s sys 0.013s
I20240502 14:13:48.024762 16544 raft_consensus.cc:3037] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.021412 16313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920ffab6592f45c795bbd8789de768a4" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe33bde6df0245cb84db702ff5a13e11"
I20240502 14:13:48.024952 16313 raft_consensus.cc:3037] T 920ffab6592f45c795bbd8789de768a4 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.025609 16515 raft_consensus.cc:348] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.025827 16515 raft_consensus.cc:374] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:48.025907 16515 raft_consensus.cc:729] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0b0185fda54117bb2b1a2f7672f386, State: Initialized, Role: FOLLOWER
I20240502 14:13:48.026194 16515 consensus_queue.cc:260] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.021893 16314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f15bd32e5a45f6bbe297bb4daaaf8a" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe33bde6df0245cb84db702ff5a13e11" is_pre_election: true
I20240502 14:13:48.021951 16432 leader_election.cc:302] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ef0b0185fda54117bb2b1a2f7672f386, fe33bde6df0245cb84db702ff5a13e11; no voters: 644249615cac43b9af8f167605f3437c
I20240502 14:13:48.027654 16512 ts_tablet_manager.cc:1398] T 920ffab6592f45c795bbd8789de768a4 P fe33bde6df0245cb84db702ff5a13e11: Time spent starting tablet: real 0.043s user 0.008s sys 0.001s
I20240502 14:13:48.027947 16512 tablet_bootstrap.cc:492] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11: Bootstrap starting.
I20240502 14:13:48.028380 16544 raft_consensus.cc:504] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.029049 16313 raft_consensus.cc:2454] T 920ffab6592f45c795bbd8789de768a4 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0b0185fda54117bb2b1a2f7672f386 in term 1.
I20240502 14:13:48.029105 16544 leader_election.cc:288] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 election: Requested vote from peers fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65:34993), 644249615cac43b9af8f167605f3437c (127.15.74.66:44425)
I20240502 14:13:48.029773 16512 tablet_bootstrap.cc:654] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:48.031513 16544 raft_consensus.cc:2781] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:13:48.031677 16544 raft_consensus.cc:482] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:48.031786 16544 raft_consensus.cc:3037] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.022785 16513 tablet_bootstrap.cc:492] T bfc19f54c7e4409a8a0ecde5be05cb48 P 644249615cac43b9af8f167605f3437c: No bootstrap required, opened a new log
I20240502 14:13:48.033324 16513 ts_tablet_manager.cc:1367] T bfc19f54c7e4409a8a0ecde5be05cb48 P 644249615cac43b9af8f167605f3437c: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.004s
I20240502 14:13:48.034366 16513 raft_consensus.cc:348] T bfc19f54c7e4409a8a0ecde5be05cb48 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.034741 16513 raft_consensus.cc:374] T bfc19f54c7e4409a8a0ecde5be05cb48 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:48.034888 16513 raft_consensus.cc:729] T bfc19f54c7e4409a8a0ecde5be05cb48 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644249615cac43b9af8f167605f3437c, State: Initialized, Role: FOLLOWER
I20240502 14:13:48.035245 16513 consensus_queue.cc:260] T bfc19f54c7e4409a8a0ecde5be05cb48 P 644249615cac43b9af8f167605f3437c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.036048 16544 raft_consensus.cc:504] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.036891 16544 leader_election.cc:288] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 election: Requested vote from peers fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65:34993), 644249615cac43b9af8f167605f3437c (127.15.74.66:44425)
I20240502 14:13:48.038038 16512 tablet_bootstrap.cc:492] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11: No bootstrap required, opened a new log
I20240502 14:13:48.038167 16512 ts_tablet_manager.cc:1367] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.003s
I20240502 14:13:48.038736 16512 raft_consensus.cc:348] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.038908 16512 raft_consensus.cc:374] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:48.038970 16512 raft_consensus.cc:729] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe33bde6df0245cb84db702ff5a13e11, State: Initialized, Role: FOLLOWER
I20240502 14:13:48.039109 16512 consensus_queue.cc:260] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.039434 16513 ts_tablet_manager.cc:1398] T bfc19f54c7e4409a8a0ecde5be05cb48 P 644249615cac43b9af8f167605f3437c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:13:48.039579 16512 ts_tablet_manager.cc:1398] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:13:48.022845 16431 leader_election.cc:302] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 644249615cac43b9af8f167605f3437c, ef0b0185fda54117bb2b1a2f7672f386; no voters:
I20240502 14:13:48.022889 16284 leader_election.cc:302] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ef0b0185fda54117bb2b1a2f7672f386, fe33bde6df0245cb84db702ff5a13e11; no voters: 644249615cac43b9af8f167605f3437c
I20240502 14:13:48.024518 16318 raft_consensus.cc:2454] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 644249615cac43b9af8f167605f3437c in term 1.
I20240502 14:13:48.041010 16391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d908982bfbc402c8892040b69d2a81c" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c"
I20240502 14:13:48.041198 16391 raft_consensus.cc:3037] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.041306 16513 tablet_bootstrap.cc:492] T eb57739bd6504281a0ab8060a69745eb P 644249615cac43b9af8f167605f3437c: Bootstrap starting.
I20240502 14:13:48.022351 16548 raft_consensus.cc:2781] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:13:48.042310 16548 raft_consensus.cc:482] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:48.042575 16548 raft_consensus.cc:3037] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.043632 16390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc19f54c7e4409a8a0ecde5be05cb48" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c"
I20240502 14:13:48.043849 16390 raft_consensus.cc:3037] T bfc19f54c7e4409a8a0ecde5be05cb48 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.021371 16523 raft_consensus.cc:504] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.021616 16517 raft_consensus.cc:482] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:48.045548 16466 raft_consensus.cc:2379] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe33bde6df0245cb84db702ff5a13e11 in current term 1: Already voted for candidate ef0b0185fda54117bb2b1a2f7672f386 in this term.
I20240502 14:13:48.045575 16517 raft_consensus.cc:504] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.045558 16546 raft_consensus.cc:2781] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:13:48.047401 16358 leader_election.cc:302] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 644249615cac43b9af8f167605f3437c, fe33bde6df0245cb84db702ff5a13e11; no voters: ef0b0185fda54117bb2b1a2f7672f386
I20240502 14:13:48.047978 16523 leader_election.cc:288] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 election: Requested vote from peers fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65:34993), 644249615cac43b9af8f167605f3437c (127.15.74.66:44425)
I20240502 14:13:48.048573 16546 raft_consensus.cc:482] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:48.048590 16391 raft_consensus.cc:2454] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0b0185fda54117bb2b1a2f7672f386 in term 1.
I20240502 14:13:48.049069 16392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80179381859d4afcb39f7c378313c9a1" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c" is_pre_election: true
I20240502 14:13:48.049099 16518 raft_consensus.cc:2781] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:48.049854 16544 raft_consensus.cc:2781] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:48.050330 16517 leader_election.cc:288] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 644249615cac43b9af8f167605f3437c (127.15.74.66:44425), ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67:43455)
I20240502 14:13:48.050385 16546 raft_consensus.cc:3037] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.050779 16392 raft_consensus.cc:2454] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe33bde6df0245cb84db702ff5a13e11 in term 0.
I20240502 14:13:48.050684 16431 leader_election.cc:302] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 644249615cac43b9af8f167605f3437c, ef0b0185fda54117bb2b1a2f7672f386; no voters:
I20240502 14:13:48.053244 16391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f15bd32e5a45f6bbe297bb4daaaf8a" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c"
I20240502 14:13:48.052604 16544 raft_consensus.cc:686] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Becoming Leader. State: Replica: ef0b0185fda54117bb2b1a2f7672f386, State: Running, Role: LEADER
I20240502 14:13:48.055999 16282 leader_election.cc:302] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 644249615cac43b9af8f167605f3437c, fe33bde6df0245cb84db702ff5a13e11; no voters:
I20240502 14:13:48.056394 16513 tablet_bootstrap.cc:654] T eb57739bd6504281a0ab8060a69745eb P 644249615cac43b9af8f167605f3437c: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:48.057336 16517 raft_consensus.cc:2781] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:13:48.057891 16515 ts_tablet_manager.cc:1398] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386: Time spent starting tablet: real 0.032s user 0.003s sys 0.013s
I20240502 14:13:48.058544 16391 raft_consensus.cc:3037] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.058758 16313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc19f54c7e4409a8a0ecde5be05cb48" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe33bde6df0245cb84db702ff5a13e11"
I20240502 14:13:48.059092 16523 raft_consensus.cc:2781] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:48.060230 16523 raft_consensus.cc:686] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Becoming Leader. State: Replica: ef0b0185fda54117bb2b1a2f7672f386, State: Running, Role: LEADER
I20240502 14:13:48.059950 16318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d908982bfbc402c8892040b69d2a81c" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe33bde6df0245cb84db702ff5a13e11"
I20240502 14:13:48.061151 16517 raft_consensus.cc:482] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:48.061290 16517 raft_consensus.cc:3037] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.063164 16517 raft_consensus.cc:504] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.064677 16518 raft_consensus.cc:686] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [term 1 LEADER]: Becoming Leader. State: Replica: 644249615cac43b9af8f167605f3437c, State: Running, Role: LEADER
I20240502 14:13:48.064957 16517 leader_election.cc:288] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 election: Requested vote from peers 644249615cac43b9af8f167605f3437c (127.15.74.66:44425), ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67:43455)
I20240502 14:13:48.065274 16392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80179381859d4afcb39f7c378313c9a1" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c"
I20240502 14:13:48.065639 16390 raft_consensus.cc:2454] T bfc19f54c7e4409a8a0ecde5be05cb48 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0b0185fda54117bb2b1a2f7672f386 in term 1.
I20240502 14:13:48.067643 16317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f15bd32e5a45f6bbe297bb4daaaf8a" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe33bde6df0245cb84db702ff5a13e11"
I20240502 14:13:48.067692 16466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80179381859d4afcb39f7c378313c9a1" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0b0185fda54117bb2b1a2f7672f386" is_pre_election: true
I20240502 14:13:48.068131 16466 raft_consensus.cc:2454] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe33bde6df0245cb84db702ff5a13e11 in term 0.
I20240502 14:13:48.068198 16523 consensus_queue.cc:237] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.068130 16391 raft_consensus.cc:2454] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0b0185fda54117bb2b1a2f7672f386 in term 1.
I20240502 14:13:48.069303 16318 raft_consensus.cc:3037] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.071651 16431 leader_election.cc:302] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 644249615cac43b9af8f167605f3437c, ef0b0185fda54117bb2b1a2f7672f386; no voters:
I20240502 14:13:48.071651 16318 raft_consensus.cc:2454] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0b0185fda54117bb2b1a2f7672f386 in term 1.
I20240502 14:13:48.071576 16548 raft_consensus.cc:504] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.072002 16549 raft_consensus.cc:2781] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:48.072190 16549 raft_consensus.cc:686] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Becoming Leader. State: Replica: ef0b0185fda54117bb2b1a2f7672f386, State: Running, Role: LEADER
I20240502 14:13:48.072734 16548 leader_election.cc:288] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 election: Requested vote from peers 644249615cac43b9af8f167605f3437c (127.15.74.66:44425), ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67:43455)
I20240502 14:13:48.072536 16549 consensus_queue.cc:237] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.073063 16391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f15bd32e5a45f6bbe297bb4daaaf8a" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c"
I20240502 14:13:48.072976 16546 raft_consensus.cc:504] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.073285 16391 raft_consensus.cc:2379] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fe33bde6df0245cb84db702ff5a13e11 in current term 1: Already voted for candidate ef0b0185fda54117bb2b1a2f7672f386 in this term.
I20240502 14:13:48.073390 16313 raft_consensus.cc:2379] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ef0b0185fda54117bb2b1a2f7672f386 in current term 1: Already voted for candidate fe33bde6df0245cb84db702ff5a13e11 in this term.
I20240502 14:13:48.073756 16546 leader_election.cc:288] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 election: Requested vote from peers 644249615cac43b9af8f167605f3437c (127.15.74.66:44425), ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67:43455)
I20240502 14:13:48.072075 16314 raft_consensus.cc:2379] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef0b0185fda54117bb2b1a2f7672f386 in current term 1: Already voted for candidate fe33bde6df0245cb84db702ff5a13e11 in this term.
I20240502 14:13:48.073882 16544 consensus_queue.cc:237] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.074370 16390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc19f54c7e4409a8a0ecde5be05cb48" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c"
I20240502 14:13:48.072034 16431 leader_election.cc:302] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 644249615cac43b9af8f167605f3437c, ef0b0185fda54117bb2b1a2f7672f386; no voters: fe33bde6df0245cb84db702ff5a13e11
I20240502 14:13:48.074636 16390 raft_consensus.cc:2379] T bfc19f54c7e4409a8a0ecde5be05cb48 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fe33bde6df0245cb84db702ff5a13e11 in current term 1: Already voted for candidate ef0b0185fda54117bb2b1a2f7672f386 in this term.
I20240502 14:13:48.074923 16550 raft_consensus.cc:2781] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:48.075394 16515 tablet_bootstrap.cc:492] T eb57739bd6504281a0ab8060a69745eb P ef0b0185fda54117bb2b1a2f7672f386: Bootstrap starting.
I20240502 14:13:48.076212 16392 raft_consensus.cc:3037] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.080355 16392 raft_consensus.cc:2454] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe33bde6df0245cb84db702ff5a13e11 in term 1.
I20240502 14:13:48.076171 16467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80179381859d4afcb39f7c378313c9a1" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0b0185fda54117bb2b1a2f7672f386"
I20240502 14:13:48.080804 16467 raft_consensus.cc:3037] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.083655 16282 leader_election.cc:302] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 644249615cac43b9af8f167605f3437c, fe33bde6df0245cb84db702ff5a13e11; no voters:
I20240502 14:13:48.084036 16546 raft_consensus.cc:2781] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:48.084230 16546 raft_consensus.cc:686] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [term 1 LEADER]: Becoming Leader. State: Replica: fe33bde6df0245cb84db702ff5a13e11, State: Running, Role: LEADER
I20240502 14:13:48.075481 16518 consensus_queue.cc:237] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.092973 16466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f15bd32e5a45f6bbe297bb4daaaf8a" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0b0185fda54117bb2b1a2f7672f386"
I20240502 14:13:48.093348 16513 tablet_bootstrap.cc:492] T eb57739bd6504281a0ab8060a69745eb P 644249615cac43b9af8f167605f3437c: No bootstrap required, opened a new log
I20240502 14:13:48.095156 16513 ts_tablet_manager.cc:1367] T eb57739bd6504281a0ab8060a69745eb P 644249615cac43b9af8f167605f3437c: Time spent bootstrapping tablet: real 0.053s user 0.010s sys 0.001s
I20240502 14:13:48.095202 16467 raft_consensus.cc:2454] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe33bde6df0245cb84db702ff5a13e11 in term 1.
I20240502 14:13:48.097259 16515 tablet_bootstrap.cc:654] T eb57739bd6504281a0ab8060a69745eb P ef0b0185fda54117bb2b1a2f7672f386: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:48.084558 16546 consensus_queue.cc:237] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.094148 16465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc19f54c7e4409a8a0ecde5be05cb48" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0b0185fda54117bb2b1a2f7672f386"
I20240502 14:13:48.101100 16513 raft_consensus.cc:348] T eb57739bd6504281a0ab8060a69745eb P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } }
I20240502 14:13:48.101437 16513 raft_consensus.cc:374] T eb57739bd6504281a0ab8060a69745eb P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:48.101545 16513 raft_consensus.cc:729] T eb57739bd6504281a0ab8060a69745eb P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644249615cac43b9af8f167605f3437c, State: Initialized, Role: FOLLOWER
I20240502 14:13:48.101866 16513 consensus_queue.cc:260] T eb57739bd6504281a0ab8060a69745eb P 644249615cac43b9af8f167605f3437c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } }
I20240502 14:13:48.102569 16513 ts_tablet_manager.cc:1398] T eb57739bd6504281a0ab8060a69745eb P 644249615cac43b9af8f167605f3437c: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 14:13:48.105443 16214 catalog_manager.cc:5484] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c reported cstate change: term changed from 0 to 1, leader changed from <none> to 644249615cac43b9af8f167605f3437c (127.15.74.66). New cstate: current_term: 1 leader_uuid: "644249615cac43b9af8f167605f3437c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } health_report { overall_health: UNKNOWN } } }
I20240502 14:13:48.110401 16515 tablet_bootstrap.cc:492] T eb57739bd6504281a0ab8060a69745eb P ef0b0185fda54117bb2b1a2f7672f386: No bootstrap required, opened a new log
I20240502 14:13:48.110611 16515 ts_tablet_manager.cc:1367] T eb57739bd6504281a0ab8060a69745eb P ef0b0185fda54117bb2b1a2f7672f386: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.012s
I20240502 14:13:48.111501 16515 raft_consensus.cc:348] T eb57739bd6504281a0ab8060a69745eb P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } }
I20240502 14:13:48.111771 16515 raft_consensus.cc:374] T eb57739bd6504281a0ab8060a69745eb P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:48.111925 16515 raft_consensus.cc:729] T eb57739bd6504281a0ab8060a69745eb P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0b0185fda54117bb2b1a2f7672f386, State: Initialized, Role: FOLLOWER
I20240502 14:13:48.112193 16550 raft_consensus.cc:686] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Becoming Leader. State: Replica: ef0b0185fda54117bb2b1a2f7672f386, State: Running, Role: LEADER
I20240502 14:13:48.112216 16515 consensus_queue.cc:260] T eb57739bd6504281a0ab8060a69745eb P ef0b0185fda54117bb2b1a2f7672f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } }
I20240502 14:13:48.112723 16550 consensus_queue.cc:237] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.113133 16515 ts_tablet_manager.cc:1398] T eb57739bd6504281a0ab8060a69745eb P ef0b0185fda54117bb2b1a2f7672f386: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:13:48.113590 16515 tablet_bootstrap.cc:492] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386: Bootstrap starting.
I20240502 14:13:48.115665 16515 tablet_bootstrap.cc:654] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:48.115512 16214 catalog_manager.cc:5484] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67). New cstate: current_term: 1 leader_uuid: "ef0b0185fda54117bb2b1a2f7672f386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:48.115325 16284 leader_election.cc:302] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe33bde6df0245cb84db702ff5a13e11; no voters: 644249615cac43b9af8f167605f3437c, ef0b0185fda54117bb2b1a2f7672f386
I20240502 14:13:48.116611 16548 raft_consensus.cc:2726] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:13:48.119668 16213 catalog_manager.cc:5484] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65). New cstate: current_term: 1 leader_uuid: "fe33bde6df0245cb84db702ff5a13e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } health_report { overall_health: UNKNOWN } } }
I20240502 14:13:48.121956 16515 tablet_bootstrap.cc:492] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386: No bootstrap required, opened a new log
I20240502 14:13:48.122141 16515 ts_tablet_manager.cc:1367] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20240502 14:13:48.122117 16284 leader_election.cc:302] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe33bde6df0245cb84db702ff5a13e11; no voters: 644249615cac43b9af8f167605f3437c, ef0b0185fda54117bb2b1a2f7672f386
I20240502 14:13:48.122597 16546 raft_consensus.cc:2726] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:13:48.122910 16515 raft_consensus.cc:348] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.123054 16515 raft_consensus.cc:374] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:48.123112 16515 raft_consensus.cc:729] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0b0185fda54117bb2b1a2f7672f386, State: Initialized, Role: FOLLOWER
I20240502 14:13:48.123306 16515 consensus_queue.cc:260] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.123734 16515 ts_tablet_manager.cc:1398] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:13:48.123937 16214 catalog_manager.cc:5484] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67). New cstate: current_term: 1 leader_uuid: "ef0b0185fda54117bb2b1a2f7672f386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:48.124266 16214 catalog_manager.cc:5484] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67). New cstate: current_term: 1 leader_uuid: "ef0b0185fda54117bb2b1a2f7672f386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:48.124610 16214 catalog_manager.cc:5484] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67). New cstate: current_term: 1 leader_uuid: "ef0b0185fda54117bb2b1a2f7672f386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:48.189710 16550 raft_consensus.cc:482] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:48.189982 16550 raft_consensus.cc:504] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.190650 16550 leader_election.cc:288] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65:34993), 644249615cac43b9af8f167605f3437c (127.15.74.66:44425)
I20240502 14:13:48.191246 16314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f5963f36815436bba21124bd3a7e9d3" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe33bde6df0245cb84db702ff5a13e11" is_pre_election: true
I20240502 14:13:48.191265 16392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f5963f36815436bba21124bd3a7e9d3" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c" is_pre_election: true
I20240502 14:13:48.191496 16314 raft_consensus.cc:2454] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0b0185fda54117bb2b1a2f7672f386 in term 0.
I20240502 14:13:48.191537 16392 raft_consensus.cc:2454] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0b0185fda54117bb2b1a2f7672f386 in term 0.
I20240502 14:13:48.191936 16432 leader_election.cc:302] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ef0b0185fda54117bb2b1a2f7672f386, fe33bde6df0245cb84db702ff5a13e11; no voters:
I20240502 14:13:48.192276 16550 raft_consensus.cc:2781] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:13:48.192375 16550 raft_consensus.cc:482] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:48.192466 16550 raft_consensus.cc:3037] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.194371 16550 raft_consensus.cc:504] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.194986 16550 leader_election.cc:288] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 election: Requested vote from peers fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65:34993), 644249615cac43b9af8f167605f3437c (127.15.74.66:44425)
I20240502 14:13:48.195438 16314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f5963f36815436bba21124bd3a7e9d3" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe33bde6df0245cb84db702ff5a13e11"
I20240502 14:13:48.195382 16392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f5963f36815436bba21124bd3a7e9d3" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c"
I20240502 14:13:48.195618 16314 raft_consensus.cc:3037] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.195623 16392 raft_consensus.cc:3037] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.198863 16392 raft_consensus.cc:2454] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0b0185fda54117bb2b1a2f7672f386 in term 1.
I20240502 14:13:48.198882 16314 raft_consensus.cc:2454] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0b0185fda54117bb2b1a2f7672f386 in term 1.
I20240502 14:13:48.199311 16431 leader_election.cc:302] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 644249615cac43b9af8f167605f3437c, ef0b0185fda54117bb2b1a2f7672f386; no voters:
I20240502 14:13:48.199628 16550 raft_consensus.cc:2781] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:48.199792 16550 raft_consensus.cc:686] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Becoming Leader. State: Replica: ef0b0185fda54117bb2b1a2f7672f386, State: Running, Role: LEADER
I20240502 14:13:48.200069 16550 consensus_queue.cc:237] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.202781 16214 catalog_manager.cc:5484] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67). New cstate: current_term: 1 leader_uuid: "ef0b0185fda54117bb2b1a2f7672f386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:48.305927 16546 raft_consensus.cc:482] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:48.306174 16546 raft_consensus.cc:504] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } }
I20240502 14:13:48.306813 16546 leader_election.cc:288] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67:43455), 644249615cac43b9af8f167605f3437c (127.15.74.66:44425)
I20240502 14:13:48.307467 16392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb57739bd6504281a0ab8060a69745eb" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c" is_pre_election: true
I20240502 14:13:48.307467 16466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb57739bd6504281a0ab8060a69745eb" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0b0185fda54117bb2b1a2f7672f386" is_pre_election: true
I20240502 14:13:48.307771 16466 raft_consensus.cc:2454] T eb57739bd6504281a0ab8060a69745eb P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe33bde6df0245cb84db702ff5a13e11 in term 0.
I20240502 14:13:48.307822 16392 raft_consensus.cc:2454] T eb57739bd6504281a0ab8060a69745eb P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe33bde6df0245cb84db702ff5a13e11 in term 0.
I20240502 14:13:48.308218 16284 leader_election.cc:302] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 644249615cac43b9af8f167605f3437c, ef0b0185fda54117bb2b1a2f7672f386, fe33bde6df0245cb84db702ff5a13e11; no voters:
I20240502 14:13:48.308516 16546 raft_consensus.cc:2781] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:13:48.308611 16546 raft_consensus.cc:482] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:48.308724 16546 raft_consensus.cc:3037] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.310642 16546 raft_consensus.cc:504] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } }
I20240502 14:13:48.311208 16546 leader_election.cc:288] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 election: Requested vote from peers ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67:43455), 644249615cac43b9af8f167605f3437c (127.15.74.66:44425)
I20240502 14:13:48.311681 16466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb57739bd6504281a0ab8060a69745eb" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0b0185fda54117bb2b1a2f7672f386"
I20240502 14:13:48.311681 16392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb57739bd6504281a0ab8060a69745eb" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c"
I20240502 14:13:48.311936 16392 raft_consensus.cc:3037] T eb57739bd6504281a0ab8060a69745eb P 644249615cac43b9af8f167605f3437c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.311985 16466 raft_consensus.cc:3037] T eb57739bd6504281a0ab8060a69745eb P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.314779 16466 raft_consensus.cc:2454] T eb57739bd6504281a0ab8060a69745eb P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe33bde6df0245cb84db702ff5a13e11 in term 1.
I20240502 14:13:48.314780 16392 raft_consensus.cc:2454] T eb57739bd6504281a0ab8060a69745eb P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe33bde6df0245cb84db702ff5a13e11 in term 1.
I20240502 14:13:48.315237 16284 leader_election.cc:302] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ef0b0185fda54117bb2b1a2f7672f386, fe33bde6df0245cb84db702ff5a13e11; no voters:
I20240502 14:13:48.315542 16546 raft_consensus.cc:2781] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:48.315722 16546 raft_consensus.cc:686] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [term 1 LEADER]: Becoming Leader. State: Replica: fe33bde6df0245cb84db702ff5a13e11, State: Running, Role: LEADER
I20240502 14:13:48.315965 16546 consensus_queue.cc:237] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } }
I20240502 14:13:48.318451 16214 catalog_manager.cc:5484] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65). New cstate: current_term: 1 leader_uuid: "fe33bde6df0245cb84db702ff5a13e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } health_report { overall_health: UNKNOWN } } }
I20240502 14:13:48.337082 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:48.340773 16565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:48.341459 16566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:48.342356 16568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:48.343096 15657 server_base.cc:1034] running on GCE node
I20240502 14:13:48.343348 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:48.343468 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:48.343524 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659228343511 us; error 0 us; skew 500 ppm
I20240502 14:13:48.343669 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:48.345085 15657 webserver.cc:460] Webserver started at http://127.15.74.68:42503/ using document root <none> and password file <none>
I20240502 14:13:48.345309 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:48.345374 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:48.345497 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:48.346189 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-3-root/instance:
uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad"
format_stamp: "Formatted at 2024-05-02 14:13:48 on dist-test-slave-wbm6"
I20240502 14:13:48.348790 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:13:48.350455 16573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:48.350824 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:13:48.350972 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-3-root
uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad"
format_stamp: "Formatted at 2024-05-02 14:13:48 on dist-test-slave-wbm6"
I20240502 14:13:48.351107 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714659223241330-15657-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:48.359628 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:48.360209 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:48.360824 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:48.361712 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:48.361802 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:48.361896 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:48.361969 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:48.368472 16550 raft_consensus.cc:482] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:48.368716 16550 raft_consensus.cc:504] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.369999 16550 leader_election.cc:288] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65:34993), 644249615cac43b9af8f167605f3437c (127.15.74.66:44425)
I20240502 14:13:48.370002 16314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42b20951bc7478d8b01231c46995aa9" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe33bde6df0245cb84db702ff5a13e11" is_pre_election: true
I20240502 14:13:48.370553 16392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42b20951bc7478d8b01231c46995aa9" candidate_uuid: "ef0b0185fda54117bb2b1a2f7672f386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644249615cac43b9af8f167605f3437c" is_pre_election: true
I20240502 14:13:48.370671 16314 raft_consensus.cc:2379] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef0b0185fda54117bb2b1a2f7672f386 in current term 1: Already voted for candidate 644249615cac43b9af8f167605f3437c in this term.
I20240502 14:13:48.371340 16432 leader_election.cc:302] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef0b0185fda54117bb2b1a2f7672f386; no voters: 644249615cac43b9af8f167605f3437c, fe33bde6df0245cb84db702ff5a13e11
I20240502 14:13:48.371846 16550 raft_consensus.cc:3037] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:48.374996 16550 raft_consensus.cc:2726] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:13:48.380467 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.68:45061
I20240502 14:13:48.380522 16635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.68:45061 every 8 connection(s)
I20240502 14:13:48.388235 16636 heartbeater.cc:347] Connected to a master server at 127.15.74.126:36099
I20240502 14:13:48.388387 16636 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:48.388840 16636 heartbeater.cc:511] Master 127.15.74.126:36099 requested a full tablet report, sending...
I20240502 14:13:48.389649 16214 ts_manager.cc:194] Registered new tserver with Master: 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68:45061)
I20240502 14:13:48.390452 16214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38224
I20240502 14:13:48.462530 16269 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:13:48.471163 16466 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "05f15bd32e5a45f6bbe297bb4daaaf8a"
dest_uuid: "ef0b0185fda54117bb2b1a2f7672f386"
mode: GRACEFUL
new_leader_uuid: "fe33bde6df0245cb84db702ff5a13e11"
from {username='slave'} at 127.0.0.1:49760
I20240502 14:13:48.471472 16466 raft_consensus.cc:595] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Received request to transfer leadership to TS fe33bde6df0245cb84db702ff5a13e11
I20240502 14:13:48.472635 16466 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d908982bfbc402c8892040b69d2a81c"
dest_uuid: "ef0b0185fda54117bb2b1a2f7672f386"
mode: GRACEFUL
new_leader_uuid: "644249615cac43b9af8f167605f3437c"
from {username='slave'} at 127.0.0.1:49760
I20240502 14:13:48.472791 16466 raft_consensus.cc:595] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Received request to transfer leadership to TS 644249615cac43b9af8f167605f3437c
I20240502 14:13:48.473676 16466 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f5963f36815436bba21124bd3a7e9d3"
dest_uuid: "ef0b0185fda54117bb2b1a2f7672f386"
mode: GRACEFUL
new_leader_uuid: "644249615cac43b9af8f167605f3437c"
from {username='slave'} at 127.0.0.1:49760
I20240502 14:13:48.473840 16466 raft_consensus.cc:595] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Received request to transfer leadership to TS 644249615cac43b9af8f167605f3437c
I20240502 14:13:48.474217 16269 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 14:13:48.474553 16269 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:13:48.475278 16550 consensus_queue.cc:1035] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.477787 16544 raft_consensus.cc:979] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386: : Instructing follower 644249615cac43b9af8f167605f3437c to start an election
I20240502 14:13:48.478037 16557 raft_consensus.cc:1067] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Signalling peer 644249615cac43b9af8f167605f3437c to start an election
I20240502 14:13:48.479100 16391 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d908982bfbc402c8892040b69d2a81c"
dest_uuid: "644249615cac43b9af8f167605f3437c"
from {username='slave'} at 127.0.0.1:34108
I20240502 14:13:48.479976 16391 raft_consensus.cc:482] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:13:48.480134 16391 raft_consensus.cc:3037] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:48.482465 16391 raft_consensus.cc:504] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.483525 16314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d908982bfbc402c8892040b69d2a81c" candidate_uuid: "644249615cac43b9af8f167605f3437c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe33bde6df0245cb84db702ff5a13e11"
I20240502 14:13:48.483528 16550 consensus_queue.cc:1035] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.483719 16466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d908982bfbc402c8892040b69d2a81c" candidate_uuid: "644249615cac43b9af8f167605f3437c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ef0b0185fda54117bb2b1a2f7672f386"
I20240502 14:13:48.483923 16466 raft_consensus.cc:3032] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:13:48.484077 16466 raft_consensus.cc:729] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ef0b0185fda54117bb2b1a2f7672f386, State: Running, Role: LEADER
I20240502 14:13:48.484437 16466 consensus_queue.cc:260] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.484701 16466 raft_consensus.cc:3037] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:48.486721 16466 raft_consensus.cc:2454] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 644249615cac43b9af8f167605f3437c in term 2.
I20240502 14:13:48.483786 16314 raft_consensus.cc:3037] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:48.487141 16359 leader_election.cc:302] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 644249615cac43b9af8f167605f3437c, ef0b0185fda54117bb2b1a2f7672f386; no voters:
I20240502 14:13:48.487461 16518 raft_consensus.cc:2781] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:13:48.487629 16518 raft_consensus.cc:686] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [term 2 LEADER]: Becoming Leader. State: Replica: 644249615cac43b9af8f167605f3437c, State: Running, Role: LEADER
I20240502 14:13:48.487955 16518 consensus_queue.cc:237] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.488248 16644 consensus_queue.cc:1035] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.489964 16314 raft_consensus.cc:2454] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 644249615cac43b9af8f167605f3437c in term 2.
I20240502 14:13:48.490296 16318 raft_consensus.cc:1226] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11 [term 2 FOLLOWER]: Rejecting Update request from peer ef0b0185fda54117bb2b1a2f7672f386 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20240502 14:13:48.491228 16214 catalog_manager.cc:5484] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c reported cstate change: term changed from 1 to 2, leader changed from ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67) to 644249615cac43b9af8f167605f3437c (127.15.74.66). New cstate: current_term: 2 leader_uuid: "644249615cac43b9af8f167605f3437c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } health_report { overall_health: UNKNOWN } } }
I20240502 14:13:48.493865 16391 leader_election.cc:288] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [CANDIDATE]: Term 2 election: Requested vote from peers fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65:34993), ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67:43455)
I20240502 14:13:48.494016 16546 consensus_queue.cc:1035] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.494097 16550 consensus_queue.cc:1035] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.495548 16644 consensus_queue.cc:1035] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.500303 16544 consensus_queue.cc:1035] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.506579 16548 consensus_queue.cc:1035] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.529449 16544 consensus_queue.cc:1035] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.533540 16550 consensus_queue.cc:1035] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.569137 16550 consensus_queue.cc:1035] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.571835 16544 raft_consensus.cc:979] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386: : Instructing follower fe33bde6df0245cb84db702ff5a13e11 to start an election
I20240502 14:13:48.572026 16549 raft_consensus.cc:1067] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Signalling peer fe33bde6df0245cb84db702ff5a13e11 to start an election
I20240502 14:13:48.572866 16314 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "05f15bd32e5a45f6bbe297bb4daaaf8a"
dest_uuid: "fe33bde6df0245cb84db702ff5a13e11"
from {username='slave'} at 127.0.0.1:37670
I20240502 14:13:48.573733 16314 raft_consensus.cc:482] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:13:48.573908 16314 raft_consensus.cc:3037] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:48.576062 16550 consensus_queue.cc:1035] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.576263 16314 raft_consensus.cc:504] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.576990 16314 leader_election.cc:288] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 2 election: Requested vote from peers 644249615cac43b9af8f167605f3437c (127.15.74.66:44425), ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67:43455)
I20240502 14:13:48.577239 16392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f15bd32e5a45f6bbe297bb4daaaf8a" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "644249615cac43b9af8f167605f3437c"
I20240502 14:13:48.578192 16465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f15bd32e5a45f6bbe297bb4daaaf8a" candidate_uuid: "fe33bde6df0245cb84db702ff5a13e11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ef0b0185fda54117bb2b1a2f7672f386"
I20240502 14:13:48.578446 16465 raft_consensus.cc:3032] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:13:48.578583 16465 raft_consensus.cc:729] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ef0b0185fda54117bb2b1a2f7672f386, State: Running, Role: LEADER
I20240502 14:13:48.578922 16465 consensus_queue.cc:260] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.579272 16465 raft_consensus.cc:3037] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:48.579349 16550 consensus_queue.cc:1035] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.582465 16465 raft_consensus.cc:2454] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe33bde6df0245cb84db702ff5a13e11 in term 2.
I20240502 14:13:48.582922 16284 leader_election.cc:302] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ef0b0185fda54117bb2b1a2f7672f386, fe33bde6df0245cb84db702ff5a13e11; no voters: 644249615cac43b9af8f167605f3437c
I20240502 14:13:48.583405 16517 raft_consensus.cc:2781] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:13:48.583581 16517 raft_consensus.cc:686] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 2 LEADER]: Becoming Leader. State: Replica: fe33bde6df0245cb84db702ff5a13e11, State: Running, Role: LEADER
I20240502 14:13:48.583850 16517 consensus_queue.cc:237] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.586625 16215 catalog_manager.cc:5484] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 reported cstate change: term changed from 1 to 2, leader changed from ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67) to fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65). New cstate: current_term: 2 leader_uuid: "fe33bde6df0245cb84db702ff5a13e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } health_report { overall_health: UNKNOWN } } }
I20240502 14:13:48.587615 16544 consensus_queue.cc:1035] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.590314 16549 raft_consensus.cc:979] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386: : Instructing follower 644249615cac43b9af8f167605f3437c to start an election
I20240502 14:13:48.590479 16549 raft_consensus.cc:1067] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Signalling peer 644249615cac43b9af8f167605f3437c to start an election
I20240502 14:13:48.591154 16391 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8f5963f36815436bba21124bd3a7e9d3"
dest_uuid: "644249615cac43b9af8f167605f3437c"
from {username='slave'} at 127.0.0.1:34108
I20240502 14:13:48.591380 16391 raft_consensus.cc:482] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:13:48.591514 16391 raft_consensus.cc:3037] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:48.593495 16391 raft_consensus.cc:504] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.594025 16391 leader_election.cc:288] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [CANDIDATE]: Term 2 election: Requested vote from peers fe33bde6df0245cb84db702ff5a13e11 (127.15.74.65:34993), ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67:43455)
I20240502 14:13:48.594461 16314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f5963f36815436bba21124bd3a7e9d3" candidate_uuid: "644249615cac43b9af8f167605f3437c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe33bde6df0245cb84db702ff5a13e11"
I20240502 14:13:48.594458 16465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f5963f36815436bba21124bd3a7e9d3" candidate_uuid: "644249615cac43b9af8f167605f3437c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ef0b0185fda54117bb2b1a2f7672f386"
I20240502 14:13:48.594673 16314 raft_consensus.cc:3037] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:48.594699 16465 raft_consensus.cc:3032] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:13:48.594769 16465 raft_consensus.cc:729] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ef0b0185fda54117bb2b1a2f7672f386, State: Running, Role: LEADER
I20240502 14:13:48.595029 16465 consensus_queue.cc:260] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.595324 16465 raft_consensus.cc:3037] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:48.596777 16314 raft_consensus.cc:2454] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 644249615cac43b9af8f167605f3437c in term 2.
I20240502 14:13:48.597167 16358 leader_election.cc:302] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 644249615cac43b9af8f167605f3437c, fe33bde6df0245cb84db702ff5a13e11; no voters:
I20240502 14:13:48.597429 16465 raft_consensus.cc:2454] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 644249615cac43b9af8f167605f3437c in term 2.
I20240502 14:13:48.597514 16644 raft_consensus.cc:2781] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:13:48.597681 16644 raft_consensus.cc:686] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [term 2 LEADER]: Becoming Leader. State: Replica: 644249615cac43b9af8f167605f3437c, State: Running, Role: LEADER
I20240502 14:13:48.597970 16644 consensus_queue.cc:237] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } }
I20240502 14:13:48.599889 16215 catalog_manager.cc:5484] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c reported cstate change: term changed from 1 to 2, leader changed from ef0b0185fda54117bb2b1a2f7672f386 (127.15.74.67) to 644249615cac43b9af8f167605f3437c (127.15.74.66). New cstate: current_term: 2 leader_uuid: "644249615cac43b9af8f167605f3437c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } health_report { overall_health: UNKNOWN } } }
I20240502 14:13:48.806746 16517 consensus_queue.cc:1035] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.810904 16546 consensus_queue.cc:1035] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.957587 16314 raft_consensus.cc:1261] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11 [term 2 FOLLOWER]: Refusing update from remote peer 644249615cac43b9af8f167605f3437c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:13:48.958145 16644 consensus_queue.cc:1035] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:48.961846 16465 raft_consensus.cc:1261] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 2 FOLLOWER]: Refusing update from remote peer 644249615cac43b9af8f167605f3437c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:13:48.962353 16552 consensus_queue.cc:1035] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:13:49.028113 16391 raft_consensus.cc:3037] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:49.029873 16465 raft_consensus.cc:1261] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 2 FOLLOWER]: Refusing update from remote peer 644249615cac43b9af8f167605f3437c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:13:49.030447 16518 consensus_queue.cc:1035] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:13:49.030611 16391 raft_consensus.cc:1261] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c [term 2 FOLLOWER]: Refusing update from remote peer fe33bde6df0245cb84db702ff5a13e11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:13:49.031144 16546 consensus_queue.cc:1035] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:13:49.033625 16314 raft_consensus.cc:1261] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11 [term 2 FOLLOWER]: Refusing update from remote peer 644249615cac43b9af8f167605f3437c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:13:49.034153 16644 consensus_queue.cc:1035] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:13:49.035655 16466 raft_consensus.cc:1261] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 2 FOLLOWER]: Refusing update from remote peer fe33bde6df0245cb84db702ff5a13e11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:13:49.036590 16546 consensus_queue.cc:1035] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:13:49.391701 16636 heartbeater.cc:503] Master 127.15.74.126:36099 was elected leader, sending a full tablet report...
I20240502 14:13:50.475104 16269 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:13:50.476490 16269 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:13:50.476632 16269 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:13:52.477013 16269 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:13:52.478266 16269 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:13:52.478432 16269 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:13:53.478451 16391 consensus_queue.cc:237] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } attrs { replace: true } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } }
I20240502 14:13:53.481879 16314 raft_consensus.cc:1261] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11 [term 2 FOLLOWER]: Refusing update from remote peer 644249615cac43b9af8f167605f3437c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:13:53.482185 16465 raft_consensus.cc:1261] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 2 FOLLOWER]: Refusing update from remote peer 644249615cac43b9af8f167605f3437c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:13:53.482946 16661 consensus_queue.cc:1035] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:13:53.484038 16661 consensus_queue.cc:1035] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:13:53.489418 16665 raft_consensus.cc:2932] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } attrs { replace: true } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } } }
I20240502 14:13:53.490178 16314 raft_consensus.cc:2932] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } attrs { replace: true } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } } }
I20240502 14:13:53.490947 16465 raft_consensus.cc:2932] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } attrs { replace: true } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } } }
I20240502 14:13:53.496796 16391 consensus_queue.cc:237] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } attrs { replace: true } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } }
I20240502 14:13:53.497100 16213 catalog_manager.cc:5484] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 reported cstate change: config changed from index -1 to 3, NON_VOTER 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68) added. New cstate: current_term: 2 leader_uuid: "644249615cac43b9af8f167605f3437c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } attrs { replace: true } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } } }
W20240502 14:13:53.499807 16356 consensus_peers.cc:485] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c -> Peer 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68:45061): Couldn't send request to peer 0a4f719afe1e4bf5afa2d652c7f8aaad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f5963f36815436bba21124bd3a7e9d3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:13:53.500360 16314 raft_consensus.cc:1261] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Refusing update from remote peer 644249615cac43b9af8f167605f3437c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:13:53.500777 16465 raft_consensus.cc:1261] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Refusing update from remote peer 644249615cac43b9af8f167605f3437c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:13:53.501139 16661 consensus_queue.cc:1035] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:13:53.500352 16356 consensus_peers.cc:485] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c -> Peer 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68:45061): Couldn't send request to peer 0a4f719afe1e4bf5afa2d652c7f8aaad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c42b20951bc7478d8b01231c46995aa9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:13:53.501528 16674 consensus_queue.cc:1035] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:13:53.505072 16661 raft_consensus.cc:2932] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } attrs { replace: true } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } } }
I20240502 14:13:53.509761 16314 raft_consensus.cc:2932] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } attrs { replace: true } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } } }
I20240502 14:13:53.510738 16215 catalog_manager.cc:5484] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c reported cstate change: config changed from index -1 to 2, NON_VOTER 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68) added. New cstate: current_term: 1 leader_uuid: "644249615cac43b9af8f167605f3437c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:13:53.512447 16465 raft_consensus.cc:2932] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } attrs { replace: true } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } } }
I20240502 14:13:53.519526 16314 consensus_queue.cc:237] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } attrs { replace: true } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } }
I20240502 14:13:53.522090 16391 raft_consensus.cc:1261] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c [term 2 FOLLOWER]: Refusing update from remote peer fe33bde6df0245cb84db702ff5a13e11: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:13:53.522812 16682 consensus_queue.cc:1035] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:13:53.522958 16465 raft_consensus.cc:1261] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 2 FOLLOWER]: Refusing update from remote peer fe33bde6df0245cb84db702ff5a13e11: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:13:53.523483 16682 consensus_queue.cc:1035] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:13:53.526741 16682 raft_consensus.cc:2932] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } attrs { replace: true } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } } }
W20240502 14:13:53.530155 16281 consensus_peers.cc:485] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 -> Peer 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68:45061): Couldn't send request to peer 0a4f719afe1e4bf5afa2d652c7f8aaad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05f15bd32e5a45f6bbe297bb4daaaf8a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:13:53.530125 16465 raft_consensus.cc:2932] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } attrs { replace: true } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } } }
I20240502 14:13:53.533095 16215 catalog_manager.cc:5484] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 reported cstate change: config changed from index -1 to 3, NON_VOTER 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68) added. New cstate: current_term: 2 leader_uuid: "fe33bde6df0245cb84db702ff5a13e11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:13:53.533293 16314 consensus_queue.cc:237] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } attrs { replace: true } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } }
I20240502 14:13:53.542563 16391 raft_consensus.cc:1261] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Refusing update from remote peer fe33bde6df0245cb84db702ff5a13e11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:13:53.542797 16281 consensus_peers.cc:485] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 -> Peer 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68:45061): Couldn't send request to peer 0a4f719afe1e4bf5afa2d652c7f8aaad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80179381859d4afcb39f7c378313c9a1. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:13:53.542903 16392 raft_consensus.cc:2932] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } attrs { replace: true } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } } }
I20240502 14:13:53.543460 16682 consensus_queue.cc:1035] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:13:53.544618 16465 raft_consensus.cc:1261] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Refusing update from remote peer fe33bde6df0245cb84db702ff5a13e11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:13:53.545222 16682 consensus_queue.cc:1035] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:13:53.546373 16687 raft_consensus.cc:2932] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } attrs { replace: true } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } } }
I20240502 14:13:53.547689 16391 raft_consensus.cc:2932] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } attrs { replace: true } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } } }
I20240502 14:13:53.552975 16214 catalog_manager.cc:5484] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c reported cstate change: config changed from index -1 to 2, NON_VOTER 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68) added. New cstate: current_term: 1 leader_uuid: "fe33bde6df0245cb84db702ff5a13e11" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } attrs { replace: true } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } } }
I20240502 14:13:53.553165 16465 raft_consensus.cc:2932] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0a4f719afe1e4bf5afa2d652c7f8aaad (127.15.74.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe33bde6df0245cb84db702ff5a13e11" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34993 } } peers { permanent_uuid: "644249615cac43b9af8f167605f3437c" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44425 } } peers { permanent_uuid: "ef0b0185fda54117bb2b1a2f7672f386" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 43455 } attrs { replace: true } } peers { permanent_uuid: "0a4f719afe1e4bf5afa2d652c7f8aaad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 45061 } attrs { promote: true } } }
I20240502 14:13:53.639243 15657 tablet_server.cc:178] TabletServer@127.15.74.65:0 shutting down...
W20240502 14:13:53.642985 16202 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20240502 14:13:53.645174 16264 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 14:13:53.645272 16264 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 14:13:53.648965 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:13:53.649266 15657 tablet_replica.cc:330] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11: stopping tablet replica
I20240502 14:13:53.649546 15657 raft_consensus.cc:2229] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:53.649855 15657 raft_consensus.cc:2258] T 8f5963f36815436bba21124bd3a7e9d3 P fe33bde6df0245cb84db702ff5a13e11 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240502 14:13:53.650246 16264 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 14:13:53.650405 16264 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 14:13:53.650699 15657 tablet_replica.cc:330] T 920ffab6592f45c795bbd8789de768a4 P fe33bde6df0245cb84db702ff5a13e11: stopping tablet replica
I20240502 14:13:53.650877 15657 raft_consensus.cc:2229] T 920ffab6592f45c795bbd8789de768a4 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:53.651072 15657 raft_consensus.cc:2258] T 920ffab6592f45c795bbd8789de768a4 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.651609 15657 tablet_replica.cc:330] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11: stopping tablet replica
I20240502 14:13:53.651762 15657 raft_consensus.cc:2229] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:53.651953 15657 raft_consensus.cc:2258] T c42b20951bc7478d8b01231c46995aa9 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.652619 15657 tablet_replica.cc:330] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11: stopping tablet replica
I20240502 14:13:53.652793 15657 raft_consensus.cc:2229] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:53.652976 15657 raft_consensus.cc:2258] T 8d908982bfbc402c8892040b69d2a81c P fe33bde6df0245cb84db702ff5a13e11 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.653484 15657 tablet_replica.cc:330] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11: stopping tablet replica
I20240502 14:13:53.653645 15657 raft_consensus.cc:2229] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:13:53.654088 15657 raft_consensus.cc:2258] T 05f15bd32e5a45f6bbe297bb4daaaf8a P fe33bde6df0245cb84db702ff5a13e11 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.654821 15657 tablet_replica.cc:330] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11: stopping tablet replica
I20240502 14:13:53.654973 15657 raft_consensus.cc:2229] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:13:53.655375 15657 raft_consensus.cc:2258] T 80179381859d4afcb39f7c378313c9a1 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.656036 15657 tablet_replica.cc:330] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11: stopping tablet replica
I20240502 14:13:53.656157 15657 raft_consensus.cc:2229] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:53.656353 15657 raft_consensus.cc:2258] T bfc19f54c7e4409a8a0ecde5be05cb48 P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.656909 15657 tablet_replica.cc:330] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11: stopping tablet replica
I20240502 14:13:53.657107 15657 raft_consensus.cc:2229] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:13:53.657390 15657 raft_consensus.cc:2258] T eb57739bd6504281a0ab8060a69745eb P fe33bde6df0245cb84db702ff5a13e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.675318 15657 tablet_server.cc:194] TabletServer@127.15.74.65:0 shutdown complete.
I20240502 14:13:53.680626 15657 tablet_server.cc:178] TabletServer@127.15.74.66:0 shutting down...
W20240502 14:13:53.683990 16264 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 14:13:53.684145 16264 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 14:13:53.686543 16264 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 14:13:53.687851 16264 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 14:13:53.690637 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:13:53.690905 15657 tablet_replica.cc:330] T eb57739bd6504281a0ab8060a69745eb P 644249615cac43b9af8f167605f3437c: stopping tablet replica
I20240502 14:13:53.691133 15657 raft_consensus.cc:2229] T eb57739bd6504281a0ab8060a69745eb P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:53.691319 15657 raft_consensus.cc:2258] T eb57739bd6504281a0ab8060a69745eb P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.691906 15657 tablet_replica.cc:330] T bfc19f54c7e4409a8a0ecde5be05cb48 P 644249615cac43b9af8f167605f3437c: stopping tablet replica
I20240502 14:13:53.692055 15657 raft_consensus.cc:2229] T bfc19f54c7e4409a8a0ecde5be05cb48 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:53.692212 15657 raft_consensus.cc:2258] T bfc19f54c7e4409a8a0ecde5be05cb48 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.692644 15657 tablet_replica.cc:330] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c: stopping tablet replica
I20240502 14:13:53.692802 15657 raft_consensus.cc:2229] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:13:53.693164 15657 raft_consensus.cc:2258] T 8f5963f36815436bba21124bd3a7e9d3 P 644249615cac43b9af8f167605f3437c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.693810 15657 tablet_replica.cc:330] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c: stopping tablet replica
I20240502 14:13:53.693959 15657 raft_consensus.cc:2229] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:53.694149 15657 raft_consensus.cc:2258] T 80179381859d4afcb39f7c378313c9a1 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.694777 15657 tablet_replica.cc:330] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c: stopping tablet replica
I20240502 14:13:53.694929 15657 raft_consensus.cc:2229] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:13:53.695230 15657 raft_consensus.cc:2258] T c42b20951bc7478d8b01231c46995aa9 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.695860 15657 tablet_replica.cc:330] T 920ffab6592f45c795bbd8789de768a4 P 644249615cac43b9af8f167605f3437c: stopping tablet replica
I20240502 14:13:53.695993 15657 raft_consensus.cc:2229] T 920ffab6592f45c795bbd8789de768a4 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:53.696141 15657 raft_consensus.cc:2258] T 920ffab6592f45c795bbd8789de768a4 P 644249615cac43b9af8f167605f3437c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.696539 15657 tablet_replica.cc:330] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c: stopping tablet replica
I20240502 14:13:53.696686 15657 raft_consensus.cc:2229] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:53.696861 15657 raft_consensus.cc:2258] T 05f15bd32e5a45f6bbe297bb4daaaf8a P 644249615cac43b9af8f167605f3437c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.697399 15657 tablet_replica.cc:330] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c: stopping tablet replica
I20240502 14:13:53.697518 15657 raft_consensus.cc:2229] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:13:53.697785 15657 raft_consensus.cc:2258] T 8d908982bfbc402c8892040b69d2a81c P 644249615cac43b9af8f167605f3437c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.713312 15657 tablet_server.cc:194] TabletServer@127.15.74.66:0 shutdown complete.
I20240502 14:13:53.717478 15657 tablet_server.cc:178] TabletServer@127.15.74.67:0 shutting down...
I20240502 14:13:53.724843 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:13:53.725124 15657 tablet_replica.cc:330] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386: stopping tablet replica
I20240502 14:13:53.725311 15657 raft_consensus.cc:2229] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:53.725523 15657 raft_consensus.cc:2258] T c42b20951bc7478d8b01231c46995aa9 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.726338 15657 tablet_replica.cc:330] T eb57739bd6504281a0ab8060a69745eb P ef0b0185fda54117bb2b1a2f7672f386: stopping tablet replica
I20240502 14:13:53.726505 15657 raft_consensus.cc:2229] T eb57739bd6504281a0ab8060a69745eb P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:53.726651 15657 raft_consensus.cc:2258] T eb57739bd6504281a0ab8060a69745eb P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.727097 15657 tablet_replica.cc:330] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386: stopping tablet replica
I20240502 14:13:53.727236 15657 raft_consensus.cc:2229] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:13:53.727584 15657 raft_consensus.cc:2258] T bfc19f54c7e4409a8a0ecde5be05cb48 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.728058 15657 tablet_replica.cc:330] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386: stopping tablet replica
I20240502 14:13:53.728227 15657 raft_consensus.cc:2229] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:53.728407 15657 raft_consensus.cc:2258] T 05f15bd32e5a45f6bbe297bb4daaaf8a P ef0b0185fda54117bb2b1a2f7672f386 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.729212 15657 tablet_replica.cc:330] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386: stopping tablet replica
I20240502 14:13:53.729372 15657 raft_consensus.cc:2229] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:53.729573 15657 raft_consensus.cc:2258] T 8d908982bfbc402c8892040b69d2a81c P ef0b0185fda54117bb2b1a2f7672f386 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.730080 15657 tablet_replica.cc:330] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386: stopping tablet replica
I20240502 14:13:53.730264 15657 raft_consensus.cc:2229] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:13:53.730602 15657 raft_consensus.cc:2258] T 920ffab6592f45c795bbd8789de768a4 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.731101 15657 tablet_replica.cc:330] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386: stopping tablet replica
I20240502 14:13:53.731247 15657 raft_consensus.cc:2229] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:53.731489 15657 raft_consensus.cc:2258] T 8f5963f36815436bba21124bd3a7e9d3 P ef0b0185fda54117bb2b1a2f7672f386 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.732290 15657 tablet_replica.cc:330] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386: stopping tablet replica
I20240502 14:13:53.732455 15657 raft_consensus.cc:2229] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:53.732671 15657 raft_consensus.cc:2258] T 80179381859d4afcb39f7c378313c9a1 P ef0b0185fda54117bb2b1a2f7672f386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.742753 15657 tablet_server.cc:194] TabletServer@127.15.74.67:0 shutdown complete.
I20240502 14:13:53.747998 15657 tablet_server.cc:178] TabletServer@127.15.74.68:0 shutting down...
I20240502 14:13:53.757083 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:13:53.770217 15657 tablet_server.cc:194] TabletServer@127.15.74.68:0 shutdown complete.
I20240502 14:13:53.773043 15657 master.cc:537] Master@127.15.74.126:36099 shutting down...
I20240502 14:13:53.780056 15657 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:13:53.780330 15657 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:53.780452 15657 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6894cdd189fe417188e31a90198edbe6: stopping tablet replica
I20240502 14:13:53.794585 15657 master.cc:552] Master@127.15.74.126:36099 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7410 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240502 14:13:53.812254 15657 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.74.126:41209,127.15.74.125:35137,127.15.74.124:40619
I20240502 14:13:53.812922 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:53.816183 16692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:53.816860 16693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:53.817734 16695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:53.818323 15657 server_base.cc:1034] running on GCE node
I20240502 14:13:53.818581 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:53.818647 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:53.818712 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659233818676 us; error 0 us; skew 500 ppm
I20240502 14:13:53.818890 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:53.820222 15657 webserver.cc:460] Webserver started at http://127.15.74.126:39643/ using document root <none> and password file <none>
I20240502 14:13:53.820458 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:53.820554 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:53.820704 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:53.821389 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-0-root/instance:
uuid: "ac0baa8b47be4d83b889ba732868021d"
format_stamp: "Formatted at 2024-05-02 14:13:53 on dist-test-slave-wbm6"
I20240502 14:13:53.823974 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:13:53.825646 16700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:53.826026 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:13:53.826125 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-0-root
uuid: "ac0baa8b47be4d83b889ba732868021d"
format_stamp: "Formatted at 2024-05-02 14:13:53 on dist-test-slave-wbm6"
I20240502 14:13:53.826239 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:53.833757 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:53.834371 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:53.848387 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.126:41209
I20240502 14:13:53.848424 16751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.126:41209 every 8 connection(s)
I20240502 14:13:53.849848 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:13:53.849934 16752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:13:53.853302 16754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:53.853622 16752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } has no permanent_uuid. Determining permanent_uuid...
W20240502 14:13:53.853909 16755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:53.854513 16757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:53.855613 15657 server_base.cc:1034] running on GCE node
I20240502 14:13:53.855904 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:53.855980 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:53.856072 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659233856030 us; error 0 us; skew 500 ppm
I20240502 14:13:53.856248 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:53.857668 15657 webserver.cc:460] Webserver started at http://127.15.74.125:39851/ using document root <none> and password file <none>
I20240502 14:13:53.857901 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:53.858006 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:53.858136 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:53.858843 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-1-root/instance:
uuid: "1377959b30e64add825a0555d3675beb"
format_stamp: "Formatted at 2024-05-02 14:13:53 on dist-test-slave-wbm6"
I20240502 14:13:53.861635 16752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:13:53.861857 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
W20240502 14:13:53.863870 16752 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.15.74.125:35137: Network error: Client connection negotiation failed: client connection to 127.15.74.125:35137: connect: Connection refused (error 111)
I20240502 14:13:53.864140 16765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:53.864518 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 14:13:53.864638 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-1-root
uuid: "1377959b30e64add825a0555d3675beb"
format_stamp: "Formatted at 2024-05-02 14:13:53 on dist-test-slave-wbm6"
I20240502 14:13:53.864777 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:53.886168 16752 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } attempt: 1
W20240502 14:13:53.888207 16752 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.15.74.125:35137: Network error: Client connection negotiation failed: client connection to 127.15.74.125:35137: connect: Connection refused (error 111)
I20240502 14:13:53.894114 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:53.894686 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:53.908383 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.125:35137
I20240502 14:13:53.908414 16816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.125:35137 every 8 connection(s)
I20240502 14:13:53.909821 16817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:53.909860 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:53.913401 16819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:53.913368 16817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } has no permanent_uuid. Determining permanent_uuid...
W20240502 14:13:53.914292 16820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:53.914912 15657 server_base.cc:1034] running on GCE node
W20240502 14:13:53.915750 16822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:53.916226 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:53.916333 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:53.916417 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659233916368 us; error 0 us; skew 500 ppm
I20240502 14:13:53.916602 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:53.918015 15657 webserver.cc:460] Webserver started at http://127.15.74.124:45893/ using document root <none> and password file <none>
I20240502 14:13:53.918248 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:53.918352 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:53.918490 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:53.919319 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-2-root/instance:
uuid: "a55ec0b90e2447febc4e3696809a97cb"
format_stamp: "Formatted at 2024-05-02 14:13:53 on dist-test-slave-wbm6"
I20240502 14:13:53.920706 16817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:13:53.925076 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 14:13:53.926975 16829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:53.927474 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:13:53.927621 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-2-root
uuid: "a55ec0b90e2447febc4e3696809a97cb"
format_stamp: "Formatted at 2024-05-02 14:13:53 on dist-test-slave-wbm6"
I20240502 14:13:53.927803 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:53.929579 16817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } has no permanent_uuid. Determining permanent_uuid...
W20240502 14:13:53.931622 16817 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.15.74.124:40619: Network error: Client connection negotiation failed: client connection to 127.15.74.124:40619: connect: Connection refused (error 111)
I20240502 14:13:53.932433 16752 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } attempt: 2
I20240502 14:13:53.936022 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:53.936762 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:53.938679 16752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } has no permanent_uuid. Determining permanent_uuid...
W20240502 14:13:53.940393 16752 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.15.74.124:40619: Network error: Client connection negotiation failed: client connection to 127.15.74.124:40619: connect: Connection refused (error 111)
I20240502 14:13:53.949846 16817 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } attempt: 1
I20240502 14:13:53.951591 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.124:40619
I20240502 14:13:53.951684 16881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.124:40619 every 8 connection(s)
W20240502 14:13:53.951855 16817 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.15.74.124:40619: Network error: Client connection negotiation failed: client connection to 127.15.74.124:40619: connect: Connection refused (error 111)
I20240502 14:13:53.952683 15657 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:13:53.953019 16882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:53.955328 16882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:13:53.961256 16882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:13:53.966498 16882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:13:53.974282 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb: Bootstrap starting.
I20240502 14:13:53.975634 16882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:53.977298 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb: No bootstrap required, opened a new log
I20240502 14:13:53.977941 16882 raft_consensus.cc:348] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } }
I20240502 14:13:53.978128 16882 raft_consensus.cc:374] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:53.978207 16882 raft_consensus.cc:729] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55ec0b90e2447febc4e3696809a97cb, State: Initialized, Role: FOLLOWER
I20240502 14:13:53.978444 16882 consensus_queue.cc:260] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } }
I20240502 14:13:53.979070 16887 sys_catalog.cc:455] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } } }
I20240502 14:13:53.979301 16887 sys_catalog.cc:458] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:13:53.979475 16882 sys_catalog.cc:564] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [sys.catalog]: configured and running, proceeding with master startup.
W20240502 14:13:53.983610 16898 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 14:13:53.983705 16898 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 14:13:53.997618 16752 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } attempt: 1
I20240502 14:13:54.005404 16752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d: Bootstrap starting.
I20240502 14:13:54.006698 16752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:54.008352 16752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d: No bootstrap required, opened a new log
I20240502 14:13:54.008955 16752 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } }
I20240502 14:13:54.009140 16752 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:54.009196 16752 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac0baa8b47be4d83b889ba732868021d, State: Initialized, Role: FOLLOWER
I20240502 14:13:54.009399 16752 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } }
I20240502 14:13:54.009959 16900 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } } }
I20240502 14:13:54.010221 16900 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:13:54.010305 16752 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:13:54.014398 15657 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240502 14:13:54.015283 16911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 14:13:54.015391 16911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 14:13:54.022047 16817 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } attempt: 2
I20240502 14:13:54.030009 16817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb: Bootstrap starting.
I20240502 14:13:54.031352 16817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:54.033061 16817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb: No bootstrap required, opened a new log
I20240502 14:13:54.033695 16817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } }
I20240502 14:13:54.033880 16817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:54.033960 16817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1377959b30e64add825a0555d3675beb, State: Initialized, Role: FOLLOWER
I20240502 14:13:54.034198 16817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } }
I20240502 14:13:54.034786 16913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } } }
I20240502 14:13:54.034968 16913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:13:54.034971 16817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:13:54.038587 15657 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240502 14:13:54.039444 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:54.039853 16924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 14:13:54.039958 16924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 14:13:54.043188 16925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:54.044126 16926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:54.044833 16928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:54.045621 15657 server_base.cc:1034] running on GCE node
I20240502 14:13:54.045882 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:54.045959 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:54.046018 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659234046002 us; error 0 us; skew 500 ppm
I20240502 14:13:54.046188 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:54.047519 15657 webserver.cc:460] Webserver started at http://127.15.74.65:39499/ using document root <none> and password file <none>
I20240502 14:13:54.047787 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:54.047868 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:54.048022 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:54.048677 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-0-root/instance:
uuid: "27c260c2306e4fc688ba829e62f8d435"
format_stamp: "Formatted at 2024-05-02 14:13:54 on dist-test-slave-wbm6"
I20240502 14:13:54.051194 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:13:54.052806 16933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:54.053216 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:13:54.053314 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-0-root
uuid: "27c260c2306e4fc688ba829e62f8d435"
format_stamp: "Formatted at 2024-05-02 14:13:54 on dist-test-slave-wbm6"
I20240502 14:13:54.053413 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:54.064141 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:54.064697 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:54.065294 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:54.066150 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:54.066227 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:54.066303 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:54.066380 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:54.081660 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.65:34965
I20240502 14:13:54.081697 16995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.65:34965 every 8 connection(s)
I20240502 14:13:54.088450 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:13:54.097337 16997 heartbeater.cc:347] Connected to a master server at 127.15.74.126:41209
I20240502 14:13:54.097515 16997 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:54.097765 16996 heartbeater.cc:347] Connected to a master server at 127.15.74.124:40619
I20240502 14:13:54.098048 16996 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:54.098685 16997 heartbeater.cc:511] Master 127.15.74.126:41209 requested a full tablet report, sending...
I20240502 14:13:54.099203 16996 heartbeater.cc:511] Master 127.15.74.124:40619 requested a full tablet report, sending...
W20240502 14:13:54.099493 17003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:54.100237 16717 ts_manager.cc:194] Registered new tserver with Master: 27c260c2306e4fc688ba829e62f8d435 (127.15.74.65:34965)
W20240502 14:13:54.100344 17004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:54.101389 16847 ts_manager.cc:194] Registered new tserver with Master: 27c260c2306e4fc688ba829e62f8d435 (127.15.74.65:34965)
I20240502 14:13:54.104729 16998 heartbeater.cc:347] Connected to a master server at 127.15.74.125:35137
I20240502 14:13:54.104943 16998 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:54.105363 16998 heartbeater.cc:511] Master 127.15.74.125:35137 requested a full tablet report, sending...
I20240502 14:13:54.106130 15657 server_base.cc:1034] running on GCE node
I20240502 14:13:54.106360 16782 ts_manager.cc:194] Registered new tserver with Master: 27c260c2306e4fc688ba829e62f8d435 (127.15.74.65:34965)
W20240502 14:13:54.106886 17007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:54.107235 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:54.107302 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:54.107357 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659234107332 us; error 0 us; skew 500 ppm
I20240502 14:13:54.107508 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:54.108781 15657 webserver.cc:460] Webserver started at http://127.15.74.66:46503/ using document root <none> and password file <none>
I20240502 14:13:54.108973 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:54.109055 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:54.109159 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:54.109791 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-1-root/instance:
uuid: "c4f39af998ec4d51987fdb5cfdabc264"
format_stamp: "Formatted at 2024-05-02 14:13:54 on dist-test-slave-wbm6"
I20240502 14:13:54.112188 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:13:54.113737 17011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:54.114078 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:13:54.114183 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-1-root
uuid: "c4f39af998ec4d51987fdb5cfdabc264"
format_stamp: "Formatted at 2024-05-02 14:13:54 on dist-test-slave-wbm6"
I20240502 14:13:54.114310 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:54.121578 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:54.122135 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:54.122872 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:54.123769 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:54.123833 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:54.123936 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:54.123991 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:54.139736 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.66:40241
I20240502 14:13:54.139765 17073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.66:40241 every 8 connection(s)
I20240502 14:13:54.143675 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:13:54.151775 17074 heartbeater.cc:347] Connected to a master server at 127.15.74.124:40619
I20240502 14:13:54.151986 17074 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:54.152657 17074 heartbeater.cc:511] Master 127.15.74.124:40619 requested a full tablet report, sending...
I20240502 14:13:54.154165 16847 ts_manager.cc:194] Registered new tserver with Master: c4f39af998ec4d51987fdb5cfdabc264 (127.15.74.66:40241)
I20240502 14:13:54.157931 17076 heartbeater.cc:347] Connected to a master server at 127.15.74.125:35137
I20240502 14:13:54.158114 17076 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:54.158722 17076 heartbeater.cc:511] Master 127.15.74.125:35137 requested a full tablet report, sending...
W20240502 14:13:54.158986 17082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:54.159229 17075 heartbeater.cc:347] Connected to a master server at 127.15.74.126:41209
I20240502 14:13:54.159358 17075 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:54.159771 16782 ts_manager.cc:194] Registered new tserver with Master: c4f39af998ec4d51987fdb5cfdabc264 (127.15.74.66:40241)
I20240502 14:13:54.159796 17075 heartbeater.cc:511] Master 127.15.74.126:41209 requested a full tablet report, sending...
W20240502 14:13:54.160693 17081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:54.161116 16717 ts_manager.cc:194] Registered new tserver with Master: c4f39af998ec4d51987fdb5cfdabc264 (127.15.74.66:40241)
I20240502 14:13:54.164155 15657 server_base.cc:1034] running on GCE node
W20240502 14:13:54.164893 17084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:54.165259 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:13:54.165347 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:13:54.165418 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659234165389 us; error 0 us; skew 500 ppm
I20240502 14:13:54.165602 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:54.166905 15657 webserver.cc:460] Webserver started at http://127.15.74.67:37829/ using document root <none> and password file <none>
I20240502 14:13:54.167143 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:54.167219 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:54.167353 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:54.167981 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-2-root/instance:
uuid: "3389635febfd4b60b8edcd2bd237e281"
format_stamp: "Formatted at 2024-05-02 14:13:54 on dist-test-slave-wbm6"
I20240502 14:13:54.170421 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:13:54.172004 17089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:54.172339 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:13:54.172454 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-2-root
uuid: "3389635febfd4b60b8edcd2bd237e281"
format_stamp: "Formatted at 2024-05-02 14:13:54 on dist-test-slave-wbm6"
I20240502 14:13:54.172560 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714659223241330-15657-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:54.186537 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:54.187168 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:54.187819 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:54.188695 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:54.188779 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:54.188874 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:54.188944 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:54.203727 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.67:42151
I20240502 14:13:54.203765 17151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.67:42151 every 8 connection(s)
I20240502 14:13:54.217531 17152 heartbeater.cc:347] Connected to a master server at 127.15.74.124:40619
I20240502 14:13:54.217715 17152 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:54.218295 17152 heartbeater.cc:511] Master 127.15.74.124:40619 requested a full tablet report, sending...
I20240502 14:13:54.219491 16847 ts_manager.cc:194] Registered new tserver with Master: 3389635febfd4b60b8edcd2bd237e281 (127.15.74.67:42151)
I20240502 14:13:54.221401 17153 heartbeater.cc:347] Connected to a master server at 127.15.74.126:41209
I20240502 14:13:54.221532 17153 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:54.222227 17153 heartbeater.cc:511] Master 127.15.74.126:41209 requested a full tablet report, sending...
I20240502 14:13:54.222548 17154 heartbeater.cc:347] Connected to a master server at 127.15.74.125:35137
I20240502 14:13:54.222676 17154 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:54.223063 17154 heartbeater.cc:511] Master 127.15.74.125:35137 requested a full tablet report, sending...
I20240502 14:13:54.223224 16717 ts_manager.cc:194] Registered new tserver with Master: 3389635febfd4b60b8edcd2bd237e281 (127.15.74.67:42151)
I20240502 14:13:54.223591 16913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:54.223760 16913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } }
I20240502 14:13:54.224259 16782 ts_manager.cc:194] Registered new tserver with Master: 3389635febfd4b60b8edcd2bd237e281 (127.15.74.67:42151)
I20240502 14:13:54.224418 16913 leader_election.cc:288] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac0baa8b47be4d83b889ba732868021d (127.15.74.126:41209), a55ec0b90e2447febc4e3696809a97cb (127.15.74.124:40619)
I20240502 14:13:54.224659 15657 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008206694s
I20240502 14:13:54.224815 16727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1377959b30e64add825a0555d3675beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac0baa8b47be4d83b889ba732868021d" is_pre_election: true
I20240502 14:13:54.225055 16727 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1377959b30e64add825a0555d3675beb in term 0.
I20240502 14:13:54.225471 16768 leader_election.cc:302] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1377959b30e64add825a0555d3675beb, ac0baa8b47be4d83b889ba732868021d; no voters:
I20240502 14:13:54.225682 16857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1377959b30e64add825a0555d3675beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55ec0b90e2447febc4e3696809a97cb" is_pre_election: true
I20240502 14:13:54.225958 16857 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1377959b30e64add825a0555d3675beb in term 0.
I20240502 14:13:54.226197 16913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:13:54.226316 16913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:54.226377 16913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:54.228395 16913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } }
I20240502 14:13:54.228920 16913 leader_election.cc:288] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [CANDIDATE]: Term 1 election: Requested vote from peers ac0baa8b47be4d83b889ba732868021d (127.15.74.126:41209), a55ec0b90e2447febc4e3696809a97cb (127.15.74.124:40619)
I20240502 14:13:54.229236 16727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1377959b30e64add825a0555d3675beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac0baa8b47be4d83b889ba732868021d"
I20240502 14:13:54.229321 16857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1377959b30e64add825a0555d3675beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55ec0b90e2447febc4e3696809a97cb"
I20240502 14:13:54.229477 16857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:54.229476 16727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:54.232472 16857 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1377959b30e64add825a0555d3675beb in term 1.
I20240502 14:13:54.232493 16727 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1377959b30e64add825a0555d3675beb in term 1.
I20240502 14:13:54.232862 16768 leader_election.cc:302] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1377959b30e64add825a0555d3675beb, a55ec0b90e2447febc4e3696809a97cb, ac0baa8b47be4d83b889ba732868021d; no voters:
I20240502 14:13:54.233134 16913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:54.233491 16913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [term 1 LEADER]: Becoming Leader. State: Replica: 1377959b30e64add825a0555d3675beb, State: Running, Role: LEADER
I20240502 14:13:54.233739 16913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } }
I20240502 14:13:54.234982 17160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1377959b30e64add825a0555d3675beb. Latest consensus state: current_term: 1 leader_uuid: "1377959b30e64add825a0555d3675beb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } } }
I20240502 14:13:54.235282 17160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:54.235914 17162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:13:54.237179 17162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:13:54.239660 16857 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 1 FOLLOWER]: Refusing update from remote peer 1377959b30e64add825a0555d3675beb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:13:54.239663 16727 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [term 1 FOLLOWER]: Refusing update from remote peer 1377959b30e64add825a0555d3675beb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:13:54.240193 16913 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:54.240410 17160 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:54.242748 16900 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1377959b30e64add825a0555d3675beb. Latest consensus state: current_term: 1 leader_uuid: "1377959b30e64add825a0555d3675beb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } } }
I20240502 14:13:54.242974 16900 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:13:54.244261 16887 sys_catalog.cc:455] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1377959b30e64add825a0555d3675beb. Latest consensus state: current_term: 1 leader_uuid: "1377959b30e64add825a0555d3675beb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } } }
I20240502 14:13:54.244462 16887 sys_catalog.cc:458] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:13:54.246047 17162 catalog_manager.cc:1347] Generated new cluster ID: f359e5d0dc0041f89a091952ba8eb7eb
I20240502 14:13:54.246156 17162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:13:54.246300 16913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1377959b30e64add825a0555d3675beb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } } }
I20240502 14:13:54.246546 16913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:54.247190 16913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1377959b30e64add825a0555d3675beb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } } }
I20240502 14:13:54.247479 16913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:54.248154 16887 sys_catalog.cc:455] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1377959b30e64add825a0555d3675beb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } } }
I20240502 14:13:54.248329 16887 sys_catalog.cc:458] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:13:54.251965 16900 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1377959b30e64add825a0555d3675beb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } } }
I20240502 14:13:54.252182 16900 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:13:54.259699 17162 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:13:54.260604 17162 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:13:54.271649 17162 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb: Generated new TSK 0
I20240502 14:13:54.271934 17162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:13:54.985082 16898 catalog_manager.cc:1259] Loaded cluster ID: f359e5d0dc0041f89a091952ba8eb7eb
I20240502 14:13:54.985219 16898 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb: loading cluster ID for follower catalog manager: success
I20240502 14:13:54.987449 16898 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb: acquiring CA information for follower catalog manager: success
I20240502 14:13:54.988466 16898 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 14:13:55.016943 16911 catalog_manager.cc:1259] Loaded cluster ID: f359e5d0dc0041f89a091952ba8eb7eb
I20240502 14:13:55.017040 16911 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d: loading cluster ID for follower catalog manager: success
I20240502 14:13:55.018890 16911 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d: acquiring CA information for follower catalog manager: success
I20240502 14:13:55.019896 16911 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 14:13:55.108619 16782 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43188
I20240502 14:13:55.161962 16782 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43190
I20240502 14:13:55.226773 16782 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43204
I20240502 14:13:56.038627 16923 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:13:56.109786 16998 heartbeater.cc:503] Master 127.15.74.125:35137 was elected leader, sending a full tablet report...
I20240502 14:13:56.163322 17076 heartbeater.cc:503] Master 127.15.74.125:35137 was elected leader, sending a full tablet report...
I20240502 14:13:56.227972 17154 heartbeater.cc:503] Master 127.15.74.125:35137 was elected leader, sending a full tablet report...
I20240502 14:13:56.384261 15657 test_util.cc:268] Using random seed: 917638640
I20240502 14:13:56.408416 16782 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43212:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 14:13:56.436407 16961 tablet_service.cc:1467] Processing CreateTablet for tablet e821ff16edf04c64851b9a63ef4cd190 (DEFAULT_TABLE table=test-workload [id=703064abe6b7459aba56f4ec61f2bf89]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 14:13:56.437031 16961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e821ff16edf04c64851b9a63ef4cd190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:56.437541 17117 tablet_service.cc:1467] Processing CreateTablet for tablet 9295ad3c5be942608a56ca4159044510 (DEFAULT_TABLE table=test-workload [id=703064abe6b7459aba56f4ec61f2bf89]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 14:13:56.438045 17117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9295ad3c5be942608a56ca4159044510. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:56.440567 17039 tablet_service.cc:1467] Processing CreateTablet for tablet d9712b737ebe4a0eb1b9a467672fe2ee (DEFAULT_TABLE table=test-workload [id=703064abe6b7459aba56f4ec61f2bf89]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 14:13:56.441009 17039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9712b737ebe4a0eb1b9a467672fe2ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:56.445065 17199 tablet_bootstrap.cc:492] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281: Bootstrap starting.
I20240502 14:13:56.447103 17199 tablet_bootstrap.cc:654] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:56.448048 17201 tablet_bootstrap.cc:492] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435: Bootstrap starting.
I20240502 14:13:56.448593 17200 tablet_bootstrap.cc:492] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264: Bootstrap starting.
I20240502 14:13:56.449807 17199 tablet_bootstrap.cc:492] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281: No bootstrap required, opened a new log
I20240502 14:13:56.449960 17199 ts_tablet_manager.cc:1367] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:13:56.450313 17200 tablet_bootstrap.cc:654] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:56.450349 17201 tablet_bootstrap.cc:654] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:56.451009 17199 raft_consensus.cc:348] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3389635febfd4b60b8edcd2bd237e281" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 42151 } }
I20240502 14:13:56.451273 17199 raft_consensus.cc:374] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:56.451505 17199 raft_consensus.cc:729] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3389635febfd4b60b8edcd2bd237e281, State: Initialized, Role: FOLLOWER
I20240502 14:13:56.451818 17199 consensus_queue.cc:260] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3389635febfd4b60b8edcd2bd237e281" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 42151 } }
I20240502 14:13:56.452057 17199 raft_consensus.cc:388] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:56.452158 17199 raft_consensus.cc:482] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:56.452287 17199 raft_consensus.cc:3037] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:56.452467 17200 tablet_bootstrap.cc:492] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264: No bootstrap required, opened a new log
I20240502 14:13:56.452612 17200 ts_tablet_manager.cc:1367] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:13:56.453444 17200 raft_consensus.cc:348] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f39af998ec4d51987fdb5cfdabc264" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 40241 } }
I20240502 14:13:56.453634 17200 raft_consensus.cc:374] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:56.453737 17200 raft_consensus.cc:729] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f39af998ec4d51987fdb5cfdabc264, State: Initialized, Role: FOLLOWER
I20240502 14:13:56.453951 17200 consensus_queue.cc:260] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f39af998ec4d51987fdb5cfdabc264" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 40241 } }
I20240502 14:13:56.454157 17200 raft_consensus.cc:388] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:56.454247 17200 raft_consensus.cc:482] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:56.454366 17200 raft_consensus.cc:3037] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:56.455261 17199 raft_consensus.cc:504] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3389635febfd4b60b8edcd2bd237e281" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 42151 } }
I20240502 14:13:56.455549 17199 leader_election.cc:302] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3389635febfd4b60b8edcd2bd237e281; no voters:
I20240502 14:13:56.456061 17199 leader_election.cc:288] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:56.456199 17205 raft_consensus.cc:2781] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:56.456739 17205 raft_consensus.cc:686] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281 [term 1 LEADER]: Becoming Leader. State: Replica: 3389635febfd4b60b8edcd2bd237e281, State: Running, Role: LEADER
I20240502 14:13:56.457105 17205 consensus_queue.cc:237] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3389635febfd4b60b8edcd2bd237e281" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 42151 } }
I20240502 14:13:56.457577 17200 raft_consensus.cc:504] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f39af998ec4d51987fdb5cfdabc264" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 40241 } }
I20240502 14:13:56.457892 17200 leader_election.cc:302] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4f39af998ec4d51987fdb5cfdabc264; no voters:
I20240502 14:13:56.458729 17209 raft_consensus.cc:2781] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:56.459123 17201 tablet_bootstrap.cc:492] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435: No bootstrap required, opened a new log
I20240502 14:13:56.459326 17201 ts_tablet_manager.cc:1367] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240502 14:13:56.459518 17200 leader_election.cc:288] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:56.461169 17201 raft_consensus.cc:348] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c260c2306e4fc688ba829e62f8d435" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34965 } }
I20240502 14:13:56.461409 17201 raft_consensus.cc:374] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:56.461539 17201 raft_consensus.cc:729] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27c260c2306e4fc688ba829e62f8d435, State: Initialized, Role: FOLLOWER
I20240502 14:13:56.461817 17201 consensus_queue.cc:260] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c260c2306e4fc688ba829e62f8d435" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34965 } }
I20240502 14:13:56.462059 17201 raft_consensus.cc:388] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:56.462147 17201 raft_consensus.cc:482] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:56.462280 17201 raft_consensus.cc:3037] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:56.464761 17199 ts_tablet_manager.cc:1398] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20240502 14:13:56.465224 17200 ts_tablet_manager.cc:1398] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20240502 14:13:56.466204 17201 raft_consensus.cc:504] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c260c2306e4fc688ba829e62f8d435" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34965 } }
I20240502 14:13:56.466463 17201 leader_election.cc:302] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27c260c2306e4fc688ba829e62f8d435; no voters:
I20240502 14:13:56.467031 17201 leader_election.cc:288] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:56.467201 17213 raft_consensus.cc:2781] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:56.469872 16782 catalog_manager.cc:5484] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3389635febfd4b60b8edcd2bd237e281 (127.15.74.67). New cstate: current_term: 1 leader_uuid: "3389635febfd4b60b8edcd2bd237e281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3389635febfd4b60b8edcd2bd237e281" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 42151 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:56.471946 17209 raft_consensus.cc:686] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264 [term 1 LEADER]: Becoming Leader. State: Replica: c4f39af998ec4d51987fdb5cfdabc264, State: Running, Role: LEADER
I20240502 14:13:56.474481 17209 consensus_queue.cc:237] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f39af998ec4d51987fdb5cfdabc264" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 40241 } }
I20240502 14:13:56.479981 17201 ts_tablet_manager.cc:1398] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435: Time spent starting tablet: real 0.020s user 0.003s sys 0.011s
I20240502 14:13:56.480465 17213 raft_consensus.cc:686] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435 [term 1 LEADER]: Becoming Leader. State: Replica: 27c260c2306e4fc688ba829e62f8d435, State: Running, Role: LEADER
I20240502 14:13:56.480885 17213 consensus_queue.cc:237] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c260c2306e4fc688ba829e62f8d435" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34965 } }
I20240502 14:13:56.484319 16781 catalog_manager.cc:5484] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27c260c2306e4fc688ba829e62f8d435 (127.15.74.65). New cstate: current_term: 1 leader_uuid: "27c260c2306e4fc688ba829e62f8d435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c260c2306e4fc688ba829e62f8d435" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 34965 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:56.490494 16782 catalog_manager.cc:5484] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4f39af998ec4d51987fdb5cfdabc264 (127.15.74.66). New cstate: current_term: 1 leader_uuid: "c4f39af998ec4d51987fdb5cfdabc264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f39af998ec4d51987fdb5cfdabc264" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 40241 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:56.508364 15657 master.cc:537] Master@127.15.74.125:35137 shutting down...
I20240502 14:13:56.514853 15657 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:13:56.515252 15657 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:56.515401 15657 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1377959b30e64add825a0555d3675beb: stopping tablet replica
I20240502 14:13:56.530326 15657 master.cc:552] Master@127.15.74.125:35137 shutdown complete.
W20240502 14:13:57.491057 17154 heartbeater.cc:644] Failed to heartbeat to 127.15.74.125:35137 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.74.125:35137: connect: Connection refused (error 111)
I20240502 14:13:57.997020 17223 raft_consensus.cc:482] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1377959b30e64add825a0555d3675beb)
I20240502 14:13:57.997226 17224 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1377959b30e64add825a0555d3675beb)
I20240502 14:13:57.997263 17223 raft_consensus.cc:504] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } }
I20240502 14:13:57.997427 17224 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } }
I20240502 14:13:57.998111 17224 leader_election.cc:288] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1377959b30e64add825a0555d3675beb (127.15.74.125:35137), a55ec0b90e2447febc4e3696809a97cb (127.15.74.124:40619)
I20240502 14:13:57.998817 16727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a55ec0b90e2447febc4e3696809a97cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ac0baa8b47be4d83b889ba732868021d" is_pre_election: true
I20240502 14:13:57.999064 16727 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a55ec0b90e2447febc4e3696809a97cb in term 1.
I20240502 14:13:57.999317 17223 leader_election.cc:288] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac0baa8b47be4d83b889ba732868021d (127.15.74.126:41209), 1377959b30e64add825a0555d3675beb (127.15.74.125:35137)
I20240502 14:13:57.999640 16832 leader_election.cc:302] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a55ec0b90e2447febc4e3696809a97cb, ac0baa8b47be4d83b889ba732868021d; no voters:
I20240502 14:13:58.000007 17223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:13:58.000155 17223 raft_consensus.cc:482] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1377959b30e64add825a0555d3675beb)
I20240502 14:13:58.000288 17223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:58.000762 16857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac0baa8b47be4d83b889ba732868021d" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "a55ec0b90e2447febc4e3696809a97cb" is_pre_election: true
W20240502 14:13:58.001482 16833 leader_election.cc:334] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1377959b30e64add825a0555d3675beb (127.15.74.125:35137): Network error: Client connection negotiation failed: client connection to 127.15.74.125:35137: connect: Connection refused (error 111)
W20240502 14:13:58.001825 16704 leader_election.cc:334] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1377959b30e64add825a0555d3675beb (127.15.74.125:35137): Network error: Client connection negotiation failed: client connection to 127.15.74.125:35137: connect: Connection refused (error 111)
I20240502 14:13:58.004106 17223 raft_consensus.cc:504] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } }
I20240502 14:13:58.004436 16857 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac0baa8b47be4d83b889ba732868021d in current term 2: Already voted for candidate a55ec0b90e2447febc4e3696809a97cb in this term.
I20240502 14:13:58.004693 17223 leader_election.cc:288] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [CANDIDATE]: Term 2 election: Requested vote from peers ac0baa8b47be4d83b889ba732868021d (127.15.74.126:41209), 1377959b30e64add825a0555d3675beb (127.15.74.125:35137)
I20240502 14:13:58.005097 16727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a55ec0b90e2447febc4e3696809a97cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ac0baa8b47be4d83b889ba732868021d"
I20240502 14:13:58.005180 16704 leader_election.cc:302] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac0baa8b47be4d83b889ba732868021d; no voters: 1377959b30e64add825a0555d3675beb, a55ec0b90e2447febc4e3696809a97cb
I20240502 14:13:58.005317 16727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:13:58.006067 16833 leader_election.cc:334] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1377959b30e64add825a0555d3675beb (127.15.74.125:35137): Network error: Client connection negotiation failed: client connection to 127.15.74.125:35137: connect: Connection refused (error 111)
I20240502 14:13:58.007253 16727 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a55ec0b90e2447febc4e3696809a97cb in term 2.
I20240502 14:13:58.007459 17224 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:13:58.007653 16832 leader_election.cc:302] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a55ec0b90e2447febc4e3696809a97cb, ac0baa8b47be4d83b889ba732868021d; no voters: 1377959b30e64add825a0555d3675beb
I20240502 14:13:58.007920 17223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:13:58.008284 17223 raft_consensus.cc:686] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 2 LEADER]: Becoming Leader. State: Replica: a55ec0b90e2447febc4e3696809a97cb, State: Running, Role: LEADER
I20240502 14:13:58.008558 17223 consensus_queue.cc:237] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } }
I20240502 14:13:58.009702 17229 sys_catalog.cc:455] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a55ec0b90e2447febc4e3696809a97cb. Latest consensus state: current_term: 2 leader_uuid: "a55ec0b90e2447febc4e3696809a97cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } } }
I20240502 14:13:58.010003 17229 sys_catalog.cc:458] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:58.010522 17231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:13:58.012264 17231 catalog_manager.cc:669] Loaded metadata for table test-workload [id=703064abe6b7459aba56f4ec61f2bf89]
I20240502 14:13:58.013676 17231 tablet_loader.cc:96] loaded metadata for tablet 9295ad3c5be942608a56ca4159044510 (table test-workload [id=703064abe6b7459aba56f4ec61f2bf89])
I20240502 14:13:58.014019 17231 tablet_loader.cc:96] loaded metadata for tablet d9712b737ebe4a0eb1b9a467672fe2ee (table test-workload [id=703064abe6b7459aba56f4ec61f2bf89])
I20240502 14:13:58.014334 17231 tablet_loader.cc:96] loaded metadata for tablet e821ff16edf04c64851b9a63ef4cd190 (table test-workload [id=703064abe6b7459aba56f4ec61f2bf89])
I20240502 14:13:58.014580 17231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:13:58.015612 17231 catalog_manager.cc:1259] Loaded cluster ID: f359e5d0dc0041f89a091952ba8eb7eb
I20240502 14:13:58.015707 17231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:13:58.016913 17231 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:13:58.017712 17231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb: Loaded TSK: 0
I20240502 14:13:58.018318 17231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:13:58.517666 16727 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [term 2 FOLLOWER]: Refusing update from remote peer a55ec0b90e2447febc4e3696809a97cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:13:58.518251 17235 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:13:58.520205 17236 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a55ec0b90e2447febc4e3696809a97cb. Latest consensus state: current_term: 2 leader_uuid: "a55ec0b90e2447febc4e3696809a97cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } } }
I20240502 14:13:58.520406 17236 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:13:58.521884 17235 sys_catalog.cc:455] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a55ec0b90e2447febc4e3696809a97cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } } }
I20240502 14:13:58.522154 17235 sys_catalog.cc:458] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:58.524215 17236 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a55ec0b90e2447febc4e3696809a97cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0baa8b47be4d83b889ba732868021d" member_type: VOTER last_known_addr { host: "127.15.74.126" port: 41209 } } peers { permanent_uuid: "1377959b30e64add825a0555d3675beb" member_type: VOTER last_known_addr { host: "127.15.74.125" port: 35137 } } peers { permanent_uuid: "a55ec0b90e2447febc4e3696809a97cb" member_type: VOTER last_known_addr { host: "127.15.74.124" port: 40619 } } }
I20240502 14:13:58.524703 17236 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [sys.catalog]: This master's current role is: FOLLOWER
W20240502 14:13:58.525159 16833 consensus_peers.cc:485] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb -> Peer 1377959b30e64add825a0555d3675beb (127.15.74.125:35137): Couldn't send request to peer 1377959b30e64add825a0555d3675beb. Status: Network error: Client connection negotiation failed: client connection to 127.15.74.125:35137: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:13:59.097407 16833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.74.125:35137: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240502 14:13:59.478713 17152 heartbeater.cc:503] Master 127.15.74.124:40619 was elected leader, sending a full tablet report...
I20240502 14:13:59.491544 16996 heartbeater.cc:503] Master 127.15.74.124:40619 was elected leader, sending a full tablet report...
I20240502 14:13:59.494148 17074 heartbeater.cc:503] Master 127.15.74.124:40619 was elected leader, sending a full tablet report...
I20240502 14:13:59.983593 16897 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:13:59.983768 16897 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:00.072712 15657 tablet_server.cc:178] TabletServer@127.15.74.65:0 shutting down...
I20240502 14:14:00.082218 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:14:00.082517 15657 tablet_replica.cc:330] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435: stopping tablet replica
I20240502 14:14:00.082721 15657 raft_consensus.cc:2229] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:00.082926 15657 raft_consensus.cc:2258] T e821ff16edf04c64851b9a63ef4cd190 P 27c260c2306e4fc688ba829e62f8d435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:00.096552 15657 tablet_server.cc:194] TabletServer@127.15.74.65:0 shutdown complete.
I20240502 14:14:00.099340 15657 tablet_server.cc:178] TabletServer@127.15.74.66:0 shutting down...
I20240502 14:14:00.107966 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:14:00.108192 15657 tablet_replica.cc:330] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264: stopping tablet replica
I20240502 14:14:00.108407 15657 raft_consensus.cc:2229] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:00.108597 15657 raft_consensus.cc:2258] T d9712b737ebe4a0eb1b9a467672fe2ee P c4f39af998ec4d51987fdb5cfdabc264 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:00.122078 15657 tablet_server.cc:194] TabletServer@127.15.74.66:0 shutdown complete.
I20240502 14:14:00.125046 15657 tablet_server.cc:178] TabletServer@127.15.74.67:0 shutting down...
I20240502 14:14:00.134052 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:14:00.134325 15657 tablet_replica.cc:330] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281: stopping tablet replica
I20240502 14:14:00.134539 15657 raft_consensus.cc:2229] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:00.134721 15657 raft_consensus.cc:2258] T 9295ad3c5be942608a56ca4159044510 P 3389635febfd4b60b8edcd2bd237e281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:00.148417 15657 tablet_server.cc:194] TabletServer@127.15.74.67:0 shutdown complete.
I20240502 14:14:00.151346 15657 master.cc:537] Master@127.15.74.126:41209 shutting down...
I20240502 14:14:00.158421 15657 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:00.158697 15657 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:00.158841 15657 tablet_replica.cc:330] T 00000000000000000000000000000000 P ac0baa8b47be4d83b889ba732868021d: stopping tablet replica
I20240502 14:14:00.172731 15657 master.cc:552] Master@127.15.74.126:41209 shutdown complete.
I20240502 14:14:00.176934 15657 master.cc:537] Master@127.15.74.124:40619 shutting down...
I20240502 14:14:00.184541 15657 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:14:00.184945 15657 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:00.185056 15657 tablet_replica.cc:330] T 00000000000000000000000000000000 P a55ec0b90e2447febc4e3696809a97cb: stopping tablet replica
I20240502 14:14:00.198526 15657 master.cc:552] Master@127.15.74.124:40619 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6398 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240502 14:14:00.210844 15657 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.74.126:36469
I20240502 14:14:00.211361 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:00.214852 17252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:00.215790 17253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:00.216480 17255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:00.217350 15657 server_base.cc:1034] running on GCE node
I20240502 14:14:00.217628 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:14:00.217691 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:14:00.217748 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659240217723 us; error 0 us; skew 500 ppm
I20240502 14:14:00.217880 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:00.223836 15657 webserver.cc:460] Webserver started at http://127.15.74.126:34959/ using document root <none> and password file <none>
I20240502 14:14:00.224049 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:00.224129 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:00.224244 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:00.224895 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/master-0-root/instance:
uuid: "20c5bdf1d0b54b7381a4c8f7043983b8"
format_stamp: "Formatted at 2024-05-02 14:14:00 on dist-test-slave-wbm6"
I20240502 14:14:00.227375 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 14:14:00.229015 17260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:00.229368 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 14:14:00.229498 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/master-0-root
uuid: "20c5bdf1d0b54b7381a4c8f7043983b8"
format_stamp: "Formatted at 2024-05-02 14:14:00 on dist-test-slave-wbm6"
I20240502 14:14:00.229651 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:00.243705 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:00.244346 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:00.258111 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.126:36469
I20240502 14:14:00.258147 17311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.126:36469 every 8 connection(s)
I20240502 14:14:00.259446 17312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:00.263381 17312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8: Bootstrap starting.
I20240502 14:14:00.264642 17312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:00.266309 17312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8: No bootstrap required, opened a new log
I20240502 14:14:00.266973 17312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c5bdf1d0b54b7381a4c8f7043983b8" member_type: VOTER }
I20240502 14:14:00.267120 17312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:00.267213 17312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c5bdf1d0b54b7381a4c8f7043983b8, State: Initialized, Role: FOLLOWER
I20240502 14:14:00.267467 17312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c5bdf1d0b54b7381a4c8f7043983b8" member_type: VOTER }
I20240502 14:14:00.267650 17312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:14:00.267733 17312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:14:00.267833 17312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:00.269593 17312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c5bdf1d0b54b7381a4c8f7043983b8" member_type: VOTER }
I20240502 14:14:00.269796 17312 leader_election.cc:302] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20c5bdf1d0b54b7381a4c8f7043983b8; no voters:
I20240502 14:14:00.270192 17312 leader_election.cc:288] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:14:00.270337 17315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:00.270836 17315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [term 1 LEADER]: Becoming Leader. State: Replica: 20c5bdf1d0b54b7381a4c8f7043983b8, State: Running, Role: LEADER
I20240502 14:14:00.271106 17312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:14:00.271126 17315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c5bdf1d0b54b7381a4c8f7043983b8" member_type: VOTER }
I20240502 14:14:00.272089 17316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20c5bdf1d0b54b7381a4c8f7043983b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c5bdf1d0b54b7381a4c8f7043983b8" member_type: VOTER } }
I20240502 14:14:00.272199 17317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20c5bdf1d0b54b7381a4c8f7043983b8. Latest consensus state: current_term: 1 leader_uuid: "20c5bdf1d0b54b7381a4c8f7043983b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c5bdf1d0b54b7381a4c8f7043983b8" member_type: VOTER } }
I20240502 14:14:00.272379 17317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:00.272361 17316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:00.276392 17320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:14:00.277999 17320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:14:00.279536 15657 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:14:00.280939 17320 catalog_manager.cc:1347] Generated new cluster ID: 2f6a1484511240189253c9206f990f52
I20240502 14:14:00.281030 17320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:14:00.310168 17320 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:14:00.310987 17320 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:14:00.321887 17320 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8: Generated new TSK 0
I20240502 14:14:00.322122 17320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:14:00.344687 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:00.348101 17333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:00.348778 17334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:00.349772 17336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:00.350255 15657 server_base.cc:1034] running on GCE node
I20240502 14:14:00.350534 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:14:00.350613 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:14:00.350675 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659240350654 us; error 0 us; skew 500 ppm
I20240502 14:14:00.350853 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:00.352180 15657 webserver.cc:460] Webserver started at http://127.15.74.65:38885/ using document root <none> and password file <none>
I20240502 14:14:00.352434 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:00.352519 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:00.352682 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:00.353386 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-0-root/instance:
uuid: "b10ad9215b7547cda6465269aa57ad15"
format_stamp: "Formatted at 2024-05-02 14:14:00 on dist-test-slave-wbm6"
I20240502 14:14:00.356482 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:14:00.358250 17341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:00.358654 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:14:00.358768 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-0-root
uuid: "b10ad9215b7547cda6465269aa57ad15"
format_stamp: "Formatted at 2024-05-02 14:14:00 on dist-test-slave-wbm6"
I20240502 14:14:00.358875 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:00.370126 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:00.370740 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:00.371331 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:00.372331 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:00.372411 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:00.372484 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:00.372537 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:00.385957 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.65:37631
I20240502 14:14:00.386015 17403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.65:37631 every 8 connection(s)
I20240502 14:14:00.387491 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:00.391616 17408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:00.392237 17409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:00.397720 17411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:00.401304 15657 server_base.cc:1034] running on GCE node
I20240502 14:14:00.401635 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:14:00.401723 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:14:00.401804 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659240401765 us; error 0 us; skew 500 ppm
I20240502 14:14:00.401968 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:00.403393 15657 webserver.cc:460] Webserver started at http://127.15.74.66:33659/ using document root <none> and password file <none>
I20240502 14:14:00.403676 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:00.403789 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:00.403930 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:00.404790 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-1-root/instance:
uuid: "18144cba013841478b210a8c9dccf277"
format_stamp: "Formatted at 2024-05-02 14:14:00 on dist-test-slave-wbm6"
I20240502 14:14:00.406085 17404 heartbeater.cc:347] Connected to a master server at 127.15.74.126:36469
I20240502 14:14:00.406220 17404 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:00.406616 17404 heartbeater.cc:511] Master 127.15.74.126:36469 requested a full tablet report, sending...
I20240502 14:14:00.407688 17277 ts_manager.cc:194] Registered new tserver with Master: b10ad9215b7547cda6465269aa57ad15 (127.15.74.65:37631)
I20240502 14:14:00.407900 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 14:14:00.408864 17277 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36364
I20240502 14:14:00.409700 17416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:00.410089 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 14:14:00.410190 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-1-root
uuid: "18144cba013841478b210a8c9dccf277"
format_stamp: "Formatted at 2024-05-02 14:14:00 on dist-test-slave-wbm6"
I20240502 14:14:00.410327 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:00.423409 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:00.423970 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:00.424468 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:00.425247 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:00.425321 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:00.425434 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:00.425490 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:00.439851 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.66:43093
I20240502 14:14:00.439879 17478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.66:43093 every 8 connection(s)
I20240502 14:14:00.441416 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:00.445242 17482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:00.446187 17483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:00.446800 17485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:00.447618 15657 server_base.cc:1034] running on GCE node
I20240502 14:14:00.448107 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:14:00.448222 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:14:00.448288 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659240448266 us; error 0 us; skew 500 ppm
I20240502 14:14:00.448480 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:00.448539 17479 heartbeater.cc:347] Connected to a master server at 127.15.74.126:36469
I20240502 14:14:00.448693 17479 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:00.449054 17479 heartbeater.cc:511] Master 127.15.74.126:36469 requested a full tablet report, sending...
I20240502 14:14:00.449919 15657 webserver.cc:460] Webserver started at http://127.15.74.67:45857/ using document root <none> and password file <none>
I20240502 14:14:00.450026 17277 ts_manager.cc:194] Registered new tserver with Master: 18144cba013841478b210a8c9dccf277 (127.15.74.66:43093)
I20240502 14:14:00.450165 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:00.450250 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:00.450403 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:00.450861 17277 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36374
I20240502 14:14:00.451064 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-2-root/instance:
uuid: "85ba3aea64414b54b177934b89a9c63f"
format_stamp: "Formatted at 2024-05-02 14:14:00 on dist-test-slave-wbm6"
I20240502 14:14:00.453765 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20240502 14:14:00.455390 17490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:00.455842 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 14:14:00.456005 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-2-root
uuid: "85ba3aea64414b54b177934b89a9c63f"
format_stamp: "Formatted at 2024-05-02 14:14:00 on dist-test-slave-wbm6"
I20240502 14:14:00.456164 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:00.478263 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:00.478761 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:00.479184 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:00.480029 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:00.480085 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:00.480137 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:00.480180 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:00.491536 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.67:45693
I20240502 14:14:00.491552 17552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.67:45693 every 8 connection(s)
I20240502 14:14:00.497912 17553 heartbeater.cc:347] Connected to a master server at 127.15.74.126:36469
I20240502 14:14:00.498030 17553 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:00.498373 17553 heartbeater.cc:511] Master 127.15.74.126:36469 requested a full tablet report, sending...
I20240502 14:14:00.499198 17277 ts_manager.cc:194] Registered new tserver with Master: 85ba3aea64414b54b177934b89a9c63f (127.15.74.67:45693)
I20240502 14:14:00.499249 15657 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006777716s
I20240502 14:14:00.500228 17277 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36390
I20240502 14:14:01.410435 17404 heartbeater.cc:503] Master 127.15.74.126:36469 was elected leader, sending a full tablet report...
I20240502 14:14:01.452122 17479 heartbeater.cc:503] Master 127.15.74.126:36469 was elected leader, sending a full tablet report...
I20240502 14:14:01.501449 17553 heartbeater.cc:503] Master 127.15.74.126:36469 was elected leader, sending a full tablet report...
I20240502 14:14:01.526757 15657 test_util.cc:268] Using random seed: 922781151
I20240502 14:14:01.538249 17277 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36394:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:14:01.538866 17277 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:14:01.557030 17518 tablet_service.cc:1467] Processing CreateTablet for tablet ac2b64044a444b968b6e895784126f1b (DEFAULT_TABLE table=test-workload [id=47bb099278a9425abe2530cf43d46e4b]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 14:14:01.557669 17518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac2b64044a444b968b6e895784126f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:01.558276 17517 tablet_service.cc:1467] Processing CreateTablet for tablet bd30b4a0e16b495fbefcd2aa4c2f3332 (DEFAULT_TABLE table=test-workload [id=47bb099278a9425abe2530cf43d46e4b]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 14:14:01.558832 17517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd30b4a0e16b495fbefcd2aa4c2f3332. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:01.571477 17573 tablet_bootstrap.cc:492] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f: Bootstrap starting.
I20240502 14:14:01.573706 17573 tablet_bootstrap.cc:654] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:01.574748 17369 tablet_service.cc:1467] Processing CreateTablet for tablet ac2b64044a444b968b6e895784126f1b (DEFAULT_TABLE table=test-workload [id=47bb099278a9425abe2530cf43d46e4b]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 14:14:01.574824 17368 tablet_service.cc:1467] Processing CreateTablet for tablet bd30b4a0e16b495fbefcd2aa4c2f3332 (DEFAULT_TABLE table=test-workload [id=47bb099278a9425abe2530cf43d46e4b]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 14:14:01.575343 17369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac2b64044a444b968b6e895784126f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:01.575572 17368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd30b4a0e16b495fbefcd2aa4c2f3332. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:01.581876 17575 tablet_bootstrap.cc:492] T bd30b4a0e16b495fbefcd2aa4c2f3332 P b10ad9215b7547cda6465269aa57ad15: Bootstrap starting.
I20240502 14:14:01.581773 17443 tablet_service.cc:1467] Processing CreateTablet for tablet bd30b4a0e16b495fbefcd2aa4c2f3332 (DEFAULT_TABLE table=test-workload [id=47bb099278a9425abe2530cf43d46e4b]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 14:14:01.582161 17573 tablet_bootstrap.cc:492] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f: No bootstrap required, opened a new log
I20240502 14:14:01.582288 17573 ts_tablet_manager.cc:1367] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240502 14:14:01.582368 17443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd30b4a0e16b495fbefcd2aa4c2f3332. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:01.583339 17573 raft_consensus.cc:348] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } }
I20240502 14:14:01.583590 17573 raft_consensus.cc:374] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:01.583628 17575 tablet_bootstrap.cc:654] T bd30b4a0e16b495fbefcd2aa4c2f3332 P b10ad9215b7547cda6465269aa57ad15: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:01.583664 17573 raft_consensus.cc:729] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ba3aea64414b54b177934b89a9c63f, State: Initialized, Role: FOLLOWER
I20240502 14:14:01.583892 17573 consensus_queue.cc:260] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } }
I20240502 14:14:01.581773 17444 tablet_service.cc:1467] Processing CreateTablet for tablet ac2b64044a444b968b6e895784126f1b (DEFAULT_TABLE table=test-workload [id=47bb099278a9425abe2530cf43d46e4b]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 14:14:01.584803 17444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac2b64044a444b968b6e895784126f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:01.584867 17573 ts_tablet_manager.cc:1398] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:01.585126 17573 tablet_bootstrap.cc:492] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f: Bootstrap starting.
I20240502 14:14:01.586592 17573 tablet_bootstrap.cc:654] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:01.592113 17579 tablet_bootstrap.cc:492] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 18144cba013841478b210a8c9dccf277: Bootstrap starting.
I20240502 14:14:01.594239 17579 tablet_bootstrap.cc:654] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 18144cba013841478b210a8c9dccf277: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:01.596516 17575 tablet_bootstrap.cc:492] T bd30b4a0e16b495fbefcd2aa4c2f3332 P b10ad9215b7547cda6465269aa57ad15: No bootstrap required, opened a new log
I20240502 14:14:01.596705 17575 ts_tablet_manager.cc:1367] T bd30b4a0e16b495fbefcd2aa4c2f3332 P b10ad9215b7547cda6465269aa57ad15: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240502 14:14:01.596856 17573 tablet_bootstrap.cc:492] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f: No bootstrap required, opened a new log
I20240502 14:14:01.596987 17573 ts_tablet_manager.cc:1367] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240502 14:14:01.597024 17579 tablet_bootstrap.cc:492] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 18144cba013841478b210a8c9dccf277: No bootstrap required, opened a new log
I20240502 14:14:01.597188 17579 ts_tablet_manager.cc:1367] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 18144cba013841478b210a8c9dccf277: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:14:01.597451 17575 raft_consensus.cc:348] T bd30b4a0e16b495fbefcd2aa4c2f3332 P b10ad9215b7547cda6465269aa57ad15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } }
I20240502 14:14:01.597548 17573 raft_consensus.cc:348] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } }
I20240502 14:14:01.597640 17575 raft_consensus.cc:374] T bd30b4a0e16b495fbefcd2aa4c2f3332 P b10ad9215b7547cda6465269aa57ad15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:01.597755 17573 raft_consensus.cc:374] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:01.597721 17579 raft_consensus.cc:348] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 18144cba013841478b210a8c9dccf277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } }
I20240502 14:14:01.597793 17575 raft_consensus.cc:729] T bd30b4a0e16b495fbefcd2aa4c2f3332 P b10ad9215b7547cda6465269aa57ad15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b10ad9215b7547cda6465269aa57ad15, State: Initialized, Role: FOLLOWER
I20240502 14:14:01.597863 17573 raft_consensus.cc:729] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ba3aea64414b54b177934b89a9c63f, State: Initialized, Role: FOLLOWER
I20240502 14:14:01.597875 17579 raft_consensus.cc:374] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 18144cba013841478b210a8c9dccf277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:01.598058 17579 raft_consensus.cc:729] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 18144cba013841478b210a8c9dccf277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18144cba013841478b210a8c9dccf277, State: Initialized, Role: FOLLOWER
I20240502 14:14:01.598088 17575 consensus_queue.cc:260] T bd30b4a0e16b495fbefcd2aa4c2f3332 P b10ad9215b7547cda6465269aa57ad15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } }
I20240502 14:14:01.598130 17573 consensus_queue.cc:260] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } }
I20240502 14:14:01.598281 17579 consensus_queue.cc:260] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 18144cba013841478b210a8c9dccf277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } }
I20240502 14:14:01.598644 17573 ts_tablet_manager.cc:1398] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:01.599321 17579 ts_tablet_manager.cc:1398] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 18144cba013841478b210a8c9dccf277: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:01.599706 17579 tablet_bootstrap.cc:492] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277: Bootstrap starting.
I20240502 14:14:01.600732 17575 ts_tablet_manager.cc:1398] T bd30b4a0e16b495fbefcd2aa4c2f3332 P b10ad9215b7547cda6465269aa57ad15: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:01.601150 17575 tablet_bootstrap.cc:492] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15: Bootstrap starting.
I20240502 14:14:01.601298 17579 tablet_bootstrap.cc:654] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:01.604132 17579 tablet_bootstrap.cc:492] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277: No bootstrap required, opened a new log
I20240502 14:14:01.604306 17579 ts_tablet_manager.cc:1367] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:14:01.604156 17575 tablet_bootstrap.cc:654] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:01.605037 17579 raft_consensus.cc:348] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } }
I20240502 14:14:01.605221 17579 raft_consensus.cc:374] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:01.605288 17579 raft_consensus.cc:729] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18144cba013841478b210a8c9dccf277, State: Initialized, Role: FOLLOWER
I20240502 14:14:01.605607 17579 consensus_queue.cc:260] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } }
I20240502 14:14:01.606072 17579 ts_tablet_manager.cc:1398] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:01.606333 17575 tablet_bootstrap.cc:492] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15: No bootstrap required, opened a new log
I20240502 14:14:01.606454 17575 ts_tablet_manager.cc:1367] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:01.606931 17575 raft_consensus.cc:348] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } }
I20240502 14:14:01.607060 17575 raft_consensus.cc:374] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:01.607122 17575 raft_consensus.cc:729] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b10ad9215b7547cda6465269aa57ad15, State: Initialized, Role: FOLLOWER
I20240502 14:14:01.607355 17575 consensus_queue.cc:260] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } }
I20240502 14:14:01.607851 17575 ts_tablet_manager.cc:1398] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:14:01.618638 17581 raft_consensus.cc:482] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:01.618779 17581 raft_consensus.cc:504] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } }
I20240502 14:14:01.619408 17581 leader_election.cc:288] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85ba3aea64414b54b177934b89a9c63f (127.15.74.67:45693), 18144cba013841478b210a8c9dccf277 (127.15.74.66:43093)
I20240502 14:14:01.626665 17454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac2b64044a444b968b6e895784126f1b" candidate_uuid: "b10ad9215b7547cda6465269aa57ad15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18144cba013841478b210a8c9dccf277" is_pre_election: true
I20240502 14:14:01.626665 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac2b64044a444b968b6e895784126f1b" candidate_uuid: "b10ad9215b7547cda6465269aa57ad15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ba3aea64414b54b177934b89a9c63f" is_pre_election: true
I20240502 14:14:01.626988 17528 raft_consensus.cc:2454] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b10ad9215b7547cda6465269aa57ad15 in term 0.
I20240502 14:14:01.626988 17454 raft_consensus.cc:2454] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b10ad9215b7547cda6465269aa57ad15 in term 0.
I20240502 14:14:01.627471 17343 leader_election.cc:302] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18144cba013841478b210a8c9dccf277, b10ad9215b7547cda6465269aa57ad15; no voters:
I20240502 14:14:01.627833 17581 raft_consensus.cc:2781] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:01.627920 17581 raft_consensus.cc:482] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:01.627990 17581 raft_consensus.cc:3037] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:01.629961 17581 raft_consensus.cc:504] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } }
I20240502 14:14:01.630522 17581 leader_election.cc:288] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [CANDIDATE]: Term 1 election: Requested vote from peers 85ba3aea64414b54b177934b89a9c63f (127.15.74.67:45693), 18144cba013841478b210a8c9dccf277 (127.15.74.66:43093)
I20240502 14:14:01.630884 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac2b64044a444b968b6e895784126f1b" candidate_uuid: "b10ad9215b7547cda6465269aa57ad15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ba3aea64414b54b177934b89a9c63f"
I20240502 14:14:01.630908 17454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac2b64044a444b968b6e895784126f1b" candidate_uuid: "b10ad9215b7547cda6465269aa57ad15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18144cba013841478b210a8c9dccf277"
I20240502 14:14:01.631052 17528 raft_consensus.cc:3037] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:01.631057 17454 raft_consensus.cc:3037] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:01.632938 17454 raft_consensus.cc:2454] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b10ad9215b7547cda6465269aa57ad15 in term 1.
I20240502 14:14:01.632987 17528 raft_consensus.cc:2454] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b10ad9215b7547cda6465269aa57ad15 in term 1.
I20240502 14:14:01.633275 17343 leader_election.cc:302] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18144cba013841478b210a8c9dccf277, b10ad9215b7547cda6465269aa57ad15; no voters:
I20240502 14:14:01.633503 17581 raft_consensus.cc:2781] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:01.633872 17581 raft_consensus.cc:686] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [term 1 LEADER]: Becoming Leader. State: Replica: b10ad9215b7547cda6465269aa57ad15, State: Running, Role: LEADER
I20240502 14:14:01.634106 17581 consensus_queue.cc:237] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } }
I20240502 14:14:01.636291 17275 catalog_manager.cc:5484] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 reported cstate change: term changed from 0 to 1, leader changed from <none> to b10ad9215b7547cda6465269aa57ad15 (127.15.74.65). New cstate: current_term: 1 leader_uuid: "b10ad9215b7547cda6465269aa57ad15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:01.731884 17576 raft_consensus.cc:482] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:01.732106 17576 raft_consensus.cc:504] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } }
I20240502 14:14:01.733067 17576 leader_election.cc:288] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b10ad9215b7547cda6465269aa57ad15 (127.15.74.65:37631), 18144cba013841478b210a8c9dccf277 (127.15.74.66:43093)
I20240502 14:14:01.740840 17379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd30b4a0e16b495fbefcd2aa4c2f3332" candidate_uuid: "85ba3aea64414b54b177934b89a9c63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b10ad9215b7547cda6465269aa57ad15" is_pre_election: true
I20240502 14:14:01.741088 17379 raft_consensus.cc:2454] T bd30b4a0e16b495fbefcd2aa4c2f3332 P b10ad9215b7547cda6465269aa57ad15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85ba3aea64414b54b177934b89a9c63f in term 0.
I20240502 14:14:01.741394 17454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd30b4a0e16b495fbefcd2aa4c2f3332" candidate_uuid: "85ba3aea64414b54b177934b89a9c63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18144cba013841478b210a8c9dccf277" is_pre_election: true
I20240502 14:14:01.741487 17491 leader_election.cc:302] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85ba3aea64414b54b177934b89a9c63f, b10ad9215b7547cda6465269aa57ad15; no voters:
I20240502 14:14:01.741559 17454 raft_consensus.cc:2454] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 18144cba013841478b210a8c9dccf277 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85ba3aea64414b54b177934b89a9c63f in term 0.
I20240502 14:14:01.741770 17576 raft_consensus.cc:2781] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:01.741847 17576 raft_consensus.cc:482] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:01.741922 17576 raft_consensus.cc:3037] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:01.743810 17576 raft_consensus.cc:504] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } }
I20240502 14:14:01.744356 17576 leader_election.cc:288] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [CANDIDATE]: Term 1 election: Requested vote from peers b10ad9215b7547cda6465269aa57ad15 (127.15.74.65:37631), 18144cba013841478b210a8c9dccf277 (127.15.74.66:43093)
I20240502 14:14:01.744761 17454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd30b4a0e16b495fbefcd2aa4c2f3332" candidate_uuid: "85ba3aea64414b54b177934b89a9c63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18144cba013841478b210a8c9dccf277"
I20240502 14:14:01.744761 17379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd30b4a0e16b495fbefcd2aa4c2f3332" candidate_uuid: "85ba3aea64414b54b177934b89a9c63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b10ad9215b7547cda6465269aa57ad15"
I20240502 14:14:01.744958 17379 raft_consensus.cc:3037] T bd30b4a0e16b495fbefcd2aa4c2f3332 P b10ad9215b7547cda6465269aa57ad15 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:01.744958 17454 raft_consensus.cc:3037] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 18144cba013841478b210a8c9dccf277 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:01.747709 17379 raft_consensus.cc:2454] T bd30b4a0e16b495fbefcd2aa4c2f3332 P b10ad9215b7547cda6465269aa57ad15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85ba3aea64414b54b177934b89a9c63f in term 1.
I20240502 14:14:01.747709 17454 raft_consensus.cc:2454] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 18144cba013841478b210a8c9dccf277 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85ba3aea64414b54b177934b89a9c63f in term 1.
I20240502 14:14:01.748131 17491 leader_election.cc:302] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85ba3aea64414b54b177934b89a9c63f, b10ad9215b7547cda6465269aa57ad15; no voters:
I20240502 14:14:01.748391 17576 raft_consensus.cc:2781] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:01.748806 17576 raft_consensus.cc:686] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [term 1 LEADER]: Becoming Leader. State: Replica: 85ba3aea64414b54b177934b89a9c63f, State: Running, Role: LEADER
I20240502 14:14:01.749068 17576 consensus_queue.cc:237] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } }
I20240502 14:14:01.751492 17275 catalog_manager.cc:5484] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f reported cstate change: term changed from 0 to 1, leader changed from <none> to 85ba3aea64414b54b177934b89a9c63f (127.15.74.67). New cstate: current_term: 1 leader_uuid: "85ba3aea64414b54b177934b89a9c63f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:02.014902 17581 consensus_queue.cc:1035] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:02.020005 17596 consensus_queue.cc:1035] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:02.279560 17330 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:02.280297 17330 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:02.280428 17330 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:02.280849 17576 consensus_queue.cc:1035] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:02.285607 17576 consensus_queue.cc:1035] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:04.280879 17330 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:04.281610 17330 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:04.281726 17330 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:06.282294 17330 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:06.283033 17330 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:06.283140 17330 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:08.283632 17330 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:08.284289 17330 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:08.284402 17330 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:10.284827 17330 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:10.285521 17330 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:10.285640 17330 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:12.286073 17330 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:12.286780 17330 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:12.286883 17330 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:12.809744 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:12.813823 17612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:12.814687 17613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:12.815526 17615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:12.816321 15657 server_base.cc:1034] running on GCE node
I20240502 14:14:12.816679 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:14:12.816745 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:14:12.816792 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659252816768 us; error 0 us; skew 500 ppm
I20240502 14:14:12.816955 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:12.818423 15657 webserver.cc:460] Webserver started at http://127.15.74.68:40715/ using document root <none> and password file <none>
I20240502 14:14:12.818675 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:12.818753 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:12.818904 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:12.819818 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-3-root/instance:
uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad"
format_stamp: "Formatted at 2024-05-02 14:14:12 on dist-test-slave-wbm6"
I20240502 14:14:12.822438 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:14:12.824100 17620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:12.824537 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 14:14:12.824638 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-3-root
uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad"
format_stamp: "Formatted at 2024-05-02 14:14:12 on dist-test-slave-wbm6"
I20240502 14:14:12.824740 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:12.848662 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:12.849323 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:12.849938 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:12.850780 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:12.850855 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:12.850982 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:12.851040 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:12.866658 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.68:34837
I20240502 14:14:12.866680 17682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.68:34837 every 8 connection(s)
I20240502 14:14:12.874431 17683 heartbeater.cc:347] Connected to a master server at 127.15.74.126:36469
I20240502 14:14:12.874581 17683 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:12.875082 17683 heartbeater.cc:511] Master 127.15.74.126:36469 requested a full tablet report, sending...
I20240502 14:14:12.876101 17276 ts_manager.cc:194] Registered new tserver with Master: f9c8e0b7fc004e95ac159d1e4fd4f0ad (127.15.74.68:34837)
I20240502 14:14:12.876942 17276 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53232
I20240502 14:14:13.302140 17379 consensus_queue.cc:237] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: true } }
I20240502 14:14:13.304862 17528 raft_consensus.cc:1261] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f [term 1 FOLLOWER]: Refusing update from remote peer b10ad9215b7547cda6465269aa57ad15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:13.305486 17609 consensus_queue.cc:1035] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:13.306150 17454 raft_consensus.cc:1261] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277 [term 1 FOLLOWER]: Refusing update from remote peer b10ad9215b7547cda6465269aa57ad15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:13.306900 17609 consensus_queue.cc:1035] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:13.308367 17690 raft_consensus.cc:2932] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f9c8e0b7fc004e95ac159d1e4fd4f0ad (127.15.74.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: true } } }
I20240502 14:14:13.308986 17528 raft_consensus.cc:2932] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f9c8e0b7fc004e95ac159d1e4fd4f0ad (127.15.74.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: true } } }
I20240502 14:14:13.313884 17276 catalog_manager.cc:5484] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f reported cstate change: config changed from index -1 to 2, NON_VOTER f9c8e0b7fc004e95ac159d1e4fd4f0ad (127.15.74.68) added. New cstate: current_term: 1 leader_uuid: "b10ad9215b7547cda6465269aa57ad15" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: true } } }
I20240502 14:14:13.314108 17454 raft_consensus.cc:2932] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f9c8e0b7fc004e95ac159d1e4fd4f0ad (127.15.74.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: true } } }
W20240502 14:14:13.314613 17344 consensus_peers.cc:485] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 -> Peer f9c8e0b7fc004e95ac159d1e4fd4f0ad (127.15.74.68:34837): Couldn't send request to peer f9c8e0b7fc004e95ac159d1e4fd4f0ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac2b64044a444b968b6e895784126f1b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:13.803328 17702 ts_tablet_manager.cc:902] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad: Initiating tablet copy from peer b10ad9215b7547cda6465269aa57ad15 (127.15.74.65:37631)
I20240502 14:14:13.805271 17702 tablet_copy_client.cc:308] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.65:37631
I20240502 14:14:13.813064 17389 tablet_copy_service.cc:140] P b10ad9215b7547cda6465269aa57ad15: Received BeginTabletCopySession request for tablet ac2b64044a444b968b6e895784126f1b from peer f9c8e0b7fc004e95ac159d1e4fd4f0ad ({username='slave'} at 127.0.0.1:44216)
I20240502 14:14:13.813388 17389 tablet_copy_service.cc:161] P b10ad9215b7547cda6465269aa57ad15: Beginning new tablet copy session on tablet ac2b64044a444b968b6e895784126f1b from peer f9c8e0b7fc004e95ac159d1e4fd4f0ad at {username='slave'} at 127.0.0.1:44216: session id = f9c8e0b7fc004e95ac159d1e4fd4f0ad-ac2b64044a444b968b6e895784126f1b
I20240502 14:14:13.816326 17389 tablet_copy_source_session.cc:205] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:13.818410 17702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac2b64044a444b968b6e895784126f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:13.824888 17702 tablet_copy_client.cc:783] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:13.825256 17702 tablet_copy_client.cc:647] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:13.827520 17702 tablet_copy_client.cc:523] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:13.831661 17702 tablet_bootstrap.cc:492] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad: Bootstrap starting.
I20240502 14:14:13.839690 17702 tablet_bootstrap.cc:492] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:13.840117 17702 tablet_bootstrap.cc:492] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad: Bootstrap complete.
I20240502 14:14:13.840466 17702 ts_tablet_manager.cc:1367] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240502 14:14:13.841266 17702 raft_consensus.cc:348] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: true } }
I20240502 14:14:13.841580 17702 raft_consensus.cc:729] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f9c8e0b7fc004e95ac159d1e4fd4f0ad, State: Initialized, Role: LEARNER
I20240502 14:14:13.841856 17702 consensus_queue.cc:260] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: true } }
I20240502 14:14:13.842602 17702 ts_tablet_manager.cc:1398] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:13.842624 17683 heartbeater.cc:503] Master 127.15.74.126:36469 was elected leader, sending a full tablet report...
I20240502 14:14:13.843540 17389 tablet_copy_service.cc:342] P b10ad9215b7547cda6465269aa57ad15: Request end of tablet copy session f9c8e0b7fc004e95ac159d1e4fd4f0ad-ac2b64044a444b968b6e895784126f1b received from {username='slave'} at 127.0.0.1:44216
I20240502 14:14:13.843678 17389 tablet_copy_service.cc:433] P b10ad9215b7547cda6465269aa57ad15: ending tablet copy session f9c8e0b7fc004e95ac159d1e4fd4f0ad-ac2b64044a444b968b6e895784126f1b on tablet ac2b64044a444b968b6e895784126f1b with peer f9c8e0b7fc004e95ac159d1e4fd4f0ad
I20240502 14:14:14.287199 17330 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:14.287801 17330 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:14.287916 17330 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:14.397640 17658 raft_consensus.cc:1203] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:14:14.967131 17712 raft_consensus.cc:1050] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15: attempting to promote NON_VOTER f9c8e0b7fc004e95ac159d1e4fd4f0ad to VOTER
I20240502 14:14:14.968041 17712 consensus_queue.cc:237] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: false } }
I20240502 14:14:14.970332 17658 raft_consensus.cc:1261] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad [term 1 LEARNER]: Refusing update from remote peer b10ad9215b7547cda6465269aa57ad15: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:14.970681 17454 raft_consensus.cc:1261] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277 [term 1 FOLLOWER]: Refusing update from remote peer b10ad9215b7547cda6465269aa57ad15: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:14.971545 17714 consensus_queue.cc:1035] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:14.972225 17714 consensus_queue.cc:1035] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:14.978597 17528 raft_consensus.cc:1261] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f [term 1 FOLLOWER]: Refusing update from remote peer b10ad9215b7547cda6465269aa57ad15: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:14.978593 17714 raft_consensus.cc:2932] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f9c8e0b7fc004e95ac159d1e4fd4f0ad (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: false } } }
I20240502 14:14:14.979188 17658 raft_consensus.cc:2932] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f9c8e0b7fc004e95ac159d1e4fd4f0ad (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: false } } }
I20240502 14:14:14.981568 17716 consensus_queue.cc:1035] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:14.983906 17528 raft_consensus.cc:2932] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f9c8e0b7fc004e95ac159d1e4fd4f0ad (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: false } } }
I20240502 14:14:14.984727 17275 catalog_manager.cc:5484] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad reported cstate change: config changed from index 2 to 3, f9c8e0b7fc004e95ac159d1e4fd4f0ad (127.15.74.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b10ad9215b7547cda6465269aa57ad15" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: false } } }
I20240502 14:14:14.986084 17454 raft_consensus.cc:2932] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f9c8e0b7fc004e95ac159d1e4fd4f0ad (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "85ba3aea64414b54b177934b89a9c63f" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: false } } }
I20240502 14:14:14.997437 17378 consensus_queue.cc:237] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: false } }
I20240502 14:14:14.999279 17658 raft_consensus.cc:1261] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad [term 1 FOLLOWER]: Refusing update from remote peer b10ad9215b7547cda6465269aa57ad15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:14.999606 17454 raft_consensus.cc:1261] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277 [term 1 FOLLOWER]: Refusing update from remote peer b10ad9215b7547cda6465269aa57ad15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:14.999790 17714 consensus_queue.cc:1035] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:15.000216 17719 consensus_queue.cc:1035] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:15.002558 17719 raft_consensus.cc:2932] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 85ba3aea64414b54b177934b89a9c63f (127.15.74.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: false } } }
I20240502 14:14:15.003235 17658 raft_consensus.cc:2932] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 85ba3aea64414b54b177934b89a9c63f (127.15.74.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: false } } }
I20240502 14:14:15.005728 17454 raft_consensus.cc:2932] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 85ba3aea64414b54b177934b89a9c63f (127.15.74.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: false } } }
I20240502 14:14:15.006765 17276 catalog_manager.cc:5484] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad reported cstate change: config changed from index 3 to 4, VOTER 85ba3aea64414b54b177934b89a9c63f (127.15.74.67) evicted. New cstate: current_term: 1 leader_uuid: "b10ad9215b7547cda6465269aa57ad15" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b10ad9215b7547cda6465269aa57ad15" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37631 } } peers { permanent_uuid: "18144cba013841478b210a8c9dccf277" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43093 } } peers { permanent_uuid: "f9c8e0b7fc004e95ac159d1e4fd4f0ad" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 34837 } attrs { promote: false } } }
I20240502 14:14:15.008255 17261 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac2b64044a444b968b6e895784126f1b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:15.011106 17517 tablet_service.cc:1514] Processing DeleteTablet for tablet ac2b64044a444b968b6e895784126f1b with delete_type TABLET_DATA_TOMBSTONED (TS 85ba3aea64414b54b177934b89a9c63f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39310
I20240502 14:14:15.012004 17727 tablet_replica.cc:330] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f: stopping tablet replica
I20240502 14:14:15.012248 17727 raft_consensus.cc:2229] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:15.012542 17727 raft_consensus.cc:2258] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:15.013468 17727 ts_tablet_manager.cc:1875] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:15.017781 17727 ts_tablet_manager.cc:1888] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:14:15.017907 17727 log.cc:1196] T ac2b64044a444b968b6e895784126f1b P 85ba3aea64414b54b177934b89a9c63f: Deleting WAL directory at /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714659223241330-15657-0/minicluster-data/ts-2-root/wals/ac2b64044a444b968b6e895784126f1b
I20240502 14:14:15.018601 17264 catalog_manager.cc:4830] TS 85ba3aea64414b54b177934b89a9c63f (127.15.74.67:45693): tablet ac2b64044a444b968b6e895784126f1b (table test-workload [id=47bb099278a9425abe2530cf43d46e4b]) successfully deleted
I20240502 14:14:16.288516 17330 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:16.289371 17330 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:16.289507 17330 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:18.290063 17330 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:18.291064 17330 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:18.291239 17330 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:20.291746 17330 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:20.292405 17330 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:20.292506 17330 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:22.293046 17330 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:22.293695 17330 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:22.293795 17330 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:24.294203 17330 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:24.294832 17330 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:24.294940 17330 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:26.295395 17330 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:26.296059 17330 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:26.296165 17330 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:26.959507 15657 tablet_server.cc:178] TabletServer@127.15.74.65:0 shutting down...
I20240502 14:14:26.971002 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:14:26.971253 15657 tablet_replica.cc:330] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15: stopping tablet replica
I20240502 14:14:26.971462 15657 raft_consensus.cc:2229] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:26.971832 15657 raft_consensus.cc:2258] T ac2b64044a444b968b6e895784126f1b P b10ad9215b7547cda6465269aa57ad15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:26.972432 15657 tablet_replica.cc:330] T bd30b4a0e16b495fbefcd2aa4c2f3332 P b10ad9215b7547cda6465269aa57ad15: stopping tablet replica
I20240502 14:14:26.972597 15657 raft_consensus.cc:2229] T bd30b4a0e16b495fbefcd2aa4c2f3332 P b10ad9215b7547cda6465269aa57ad15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:26.972749 15657 raft_consensus.cc:2258] T bd30b4a0e16b495fbefcd2aa4c2f3332 P b10ad9215b7547cda6465269aa57ad15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:26.986811 15657 tablet_server.cc:194] TabletServer@127.15.74.65:0 shutdown complete.
I20240502 14:14:26.989946 15657 tablet_server.cc:178] TabletServer@127.15.74.66:0 shutting down...
I20240502 14:14:26.998574 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:14:26.998823 15657 tablet_replica.cc:330] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277: stopping tablet replica
I20240502 14:14:26.999019 15657 raft_consensus.cc:2229] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:26.999258 15657 raft_consensus.cc:2258] T ac2b64044a444b968b6e895784126f1b P 18144cba013841478b210a8c9dccf277 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:26.999859 15657 tablet_replica.cc:330] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 18144cba013841478b210a8c9dccf277: stopping tablet replica
I20240502 14:14:26.999998 15657 raft_consensus.cc:2229] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 18144cba013841478b210a8c9dccf277 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:27.000141 15657 raft_consensus.cc:2258] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 18144cba013841478b210a8c9dccf277 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:27.016034 15657 tablet_server.cc:194] TabletServer@127.15.74.66:0 shutdown complete.
I20240502 14:14:27.019150 15657 tablet_server.cc:178] TabletServer@127.15.74.67:0 shutting down...
I20240502 14:14:27.029917 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:14:27.030257 15657 tablet_replica.cc:330] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f: stopping tablet replica
I20240502 14:14:27.030439 15657 raft_consensus.cc:2229] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:27.030771 15657 raft_consensus.cc:2258] T bd30b4a0e16b495fbefcd2aa4c2f3332 P 85ba3aea64414b54b177934b89a9c63f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:27.044618 15657 tablet_server.cc:194] TabletServer@127.15.74.67:0 shutdown complete.
I20240502 14:14:27.047485 15657 tablet_server.cc:178] TabletServer@127.15.74.68:0 shutting down...
I20240502 14:14:27.056258 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:14:27.056514 15657 tablet_replica.cc:330] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad: stopping tablet replica
I20240502 14:14:27.056708 15657 raft_consensus.cc:2229] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:27.056922 15657 raft_consensus.cc:2258] T ac2b64044a444b968b6e895784126f1b P f9c8e0b7fc004e95ac159d1e4fd4f0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:27.070140 15657 tablet_server.cc:194] TabletServer@127.15.74.68:0 shutdown complete.
I20240502 14:14:27.073035 15657 master.cc:537] Master@127.15.74.126:36469 shutting down...
I20240502 14:14:27.086901 15657 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:27.087168 15657 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:27.087409 15657 tablet_replica.cc:330] T 00000000000000000000000000000000 P 20c5bdf1d0b54b7381a4c8f7043983b8: stopping tablet replica
I20240502 14:14:27.100767 15657 master.cc:552] Master@127.15.74.126:36469 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26901 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240502 14:14:27.112342 15657 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.74.126:38137
I20240502 14:14:27.112847 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:27.115726 17732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:27.116679 17733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:27.117463 17735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:27.118345 15657 server_base.cc:1034] running on GCE node
I20240502 14:14:27.118558 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:14:27.118603 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:14:27.118628 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659267118624 us; error 0 us; skew 500 ppm
I20240502 14:14:27.118785 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:27.124454 15657 webserver.cc:460] Webserver started at http://127.15.74.126:45593/ using document root <none> and password file <none>
I20240502 14:14:27.124693 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:27.124756 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:27.124879 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:27.125517 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714659223241330-15657-0/minicluster-data/master-0-root/instance:
uuid: "486658398b8e4d56924b7e5b4ee15960"
format_stamp: "Formatted at 2024-05-02 14:14:27 on dist-test-slave-wbm6"
I20240502 14:14:27.128129 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:27.130045 17740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:27.130460 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:14:27.130584 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714659223241330-15657-0/minicluster-data/master-0-root
uuid: "486658398b8e4d56924b7e5b4ee15960"
format_stamp: "Formatted at 2024-05-02 14:14:27 on dist-test-slave-wbm6"
I20240502 14:14:27.130705 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714659223241330-15657-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714659223241330-15657-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714659223241330-15657-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:27.136464 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:27.136960 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:27.146739 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.126:38137
I20240502 14:14:27.146754 17791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.126:38137 every 8 connection(s)
I20240502 14:14:27.147979 17792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:27.151952 17792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960: Bootstrap starting.
I20240502 14:14:27.152987 17792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:27.154273 17792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960: No bootstrap required, opened a new log
I20240502 14:14:27.154801 17792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486658398b8e4d56924b7e5b4ee15960" member_type: VOTER }
I20240502 14:14:27.154911 17792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:27.154961 17792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486658398b8e4d56924b7e5b4ee15960, State: Initialized, Role: FOLLOWER
I20240502 14:14:27.155130 17792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486658398b8e4d56924b7e5b4ee15960" member_type: VOTER }
I20240502 14:14:27.155222 17792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:14:27.155275 17792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:14:27.155349 17792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:27.157058 17792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486658398b8e4d56924b7e5b4ee15960" member_type: VOTER }
I20240502 14:14:27.157231 17792 leader_election.cc:302] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 486658398b8e4d56924b7e5b4ee15960; no voters:
I20240502 14:14:27.157640 17792 leader_election.cc:288] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:14:27.157752 17795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:27.158221 17795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [term 1 LEADER]: Becoming Leader. State: Replica: 486658398b8e4d56924b7e5b4ee15960, State: Running, Role: LEADER
I20240502 14:14:27.158440 17792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:14:27.158450 17795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486658398b8e4d56924b7e5b4ee15960" member_type: VOTER }
I20240502 14:14:27.159850 17797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 486658398b8e4d56924b7e5b4ee15960. Latest consensus state: current_term: 1 leader_uuid: "486658398b8e4d56924b7e5b4ee15960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486658398b8e4d56924b7e5b4ee15960" member_type: VOTER } }
I20240502 14:14:27.160004 17797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:27.163535 17796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "486658398b8e4d56924b7e5b4ee15960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486658398b8e4d56924b7e5b4ee15960" member_type: VOTER } }
I20240502 14:14:27.163678 17796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:27.164239 17803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:14:27.164701 15657 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:14:27.165485 17803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:14:27.168061 17803 catalog_manager.cc:1347] Generated new cluster ID: fe7270022f57425181470652b7602690
I20240502 14:14:27.168138 17803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:14:27.174255 17803 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:14:27.174863 17803 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:14:27.184793 17803 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960: Generated new TSK 0
I20240502 14:14:27.184969 17803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:14:27.196808 15657 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 7.6014e-05s
I20240502 14:14:39.283171 15657 master.cc:537] Master@127.15.74.126:38137 shutting down...
I20240502 14:14:39.295755 15657 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:39.295977 15657 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:39.296049 15657 tablet_replica.cc:330] T 00000000000000000000000000000000 P 486658398b8e4d56924b7e5b4ee15960: stopping tablet replica
I20240502 14:14:39.315848 15657 master.cc:552] Master@127.15.74.126:38137 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12211 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240502 14:14:39.323709 15657 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.74.126:35925
I20240502 14:14:39.324152 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:39.326443 17814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:39.327059 17815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:39.327644 17817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:39.327903 15657 server_base.cc:1034] running on GCE node
I20240502 14:14:39.328133 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:14:39.328193 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:14:39.328253 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659279328233 us; error 0 us; skew 500 ppm
I20240502 14:14:39.328416 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:39.329602 15657 webserver.cc:460] Webserver started at http://127.15.74.126:38065/ using document root <none> and password file <none>
I20240502 14:14:39.329828 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:39.329905 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:39.330050 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:39.330698 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/master-0-root/instance:
uuid: "bdd24413df134a4bab21c580d072fff2"
format_stamp: "Formatted at 2024-05-02 14:14:39 on dist-test-slave-wbm6"
I20240502 14:14:39.333232 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:39.334599 17822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:39.334905 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:14:39.335012 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/master-0-root
uuid: "bdd24413df134a4bab21c580d072fff2"
format_stamp: "Formatted at 2024-05-02 14:14:39 on dist-test-slave-wbm6"
I20240502 14:14:39.335129 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:39.346560 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:39.347069 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:39.357126 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.126:35925
I20240502 14:14:39.357149 17873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.126:35925 every 8 connection(s)
I20240502 14:14:39.358300 17874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.362226 17874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2: Bootstrap starting.
I20240502 14:14:39.363220 17874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.364535 17874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2: No bootstrap required, opened a new log
I20240502 14:14:39.365110 17874 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd24413df134a4bab21c580d072fff2" member_type: VOTER }
I20240502 14:14:39.365238 17874 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.365320 17874 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd24413df134a4bab21c580d072fff2, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.365515 17874 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd24413df134a4bab21c580d072fff2" member_type: VOTER }
I20240502 14:14:39.365883 17874 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:14:39.366003 17874 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:14:39.366112 17874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.367924 17874 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd24413df134a4bab21c580d072fff2" member_type: VOTER }
I20240502 14:14:39.368125 17874 leader_election.cc:302] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdd24413df134a4bab21c580d072fff2; no voters:
I20240502 14:14:39.368503 17874 leader_election.cc:288] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:14:39.368623 17877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:39.369089 17877 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [term 1 LEADER]: Becoming Leader. State: Replica: bdd24413df134a4bab21c580d072fff2, State: Running, Role: LEADER
I20240502 14:14:39.369362 17874 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:14:39.369334 17877 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd24413df134a4bab21c580d072fff2" member_type: VOTER }
I20240502 14:14:39.370105 17879 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdd24413df134a4bab21c580d072fff2. Latest consensus state: current_term: 1 leader_uuid: "bdd24413df134a4bab21c580d072fff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd24413df134a4bab21c580d072fff2" member_type: VOTER } }
I20240502 14:14:39.370164 17878 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdd24413df134a4bab21c580d072fff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd24413df134a4bab21c580d072fff2" member_type: VOTER } }
I20240502 14:14:39.370311 17879 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:39.370338 17878 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:39.371520 17881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:14:39.372808 17881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:14:39.374686 15657 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:14:39.375636 17881 catalog_manager.cc:1347] Generated new cluster ID: 0eb5b91d490d49d688f065042a2299ee
I20240502 14:14:39.375717 17881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:14:39.385277 17881 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:14:39.385937 17881 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:14:39.406721 17881 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2: Generated new TSK 0
I20240502 14:14:39.406991 17881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:14:39.440084 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:39.442934 17895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:39.443651 17896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:39.443775 17898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:39.444447 15657 server_base.cc:1034] running on GCE node
I20240502 14:14:39.444736 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:14:39.444816 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:14:39.444875 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659279444856 us; error 0 us; skew 500 ppm
I20240502 14:14:39.445032 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:39.446300 15657 webserver.cc:460] Webserver started at http://127.15.74.65:41773/ using document root <none> and password file <none>
I20240502 14:14:39.446580 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:39.446681 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:39.446820 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:39.447551 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/ts-0-root/instance:
uuid: "a897be9787a44b08aa7b722d260c94c6"
format_stamp: "Formatted at 2024-05-02 14:14:39 on dist-test-slave-wbm6"
I20240502 14:14:39.450011 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:39.451527 17903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:39.451915 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:14:39.452016 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/ts-0-root
uuid: "a897be9787a44b08aa7b722d260c94c6"
format_stamp: "Formatted at 2024-05-02 14:14:39 on dist-test-slave-wbm6"
I20240502 14:14:39.452116 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:39.473160 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:39.473676 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:39.474125 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:39.475014 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:39.475082 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:39.475179 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:39.475243 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:39.487252 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.65:43309
I20240502 14:14:39.487285 17965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.65:43309 every 8 connection(s)
I20240502 14:14:39.488955 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:39.492342 17970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:39.492923 17971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:39.494740 17973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:39.495527 15657 server_base.cc:1034] running on GCE node
I20240502 14:14:39.495846 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:14:39.495940 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:14:39.496001 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659279495973 us; error 0 us; skew 500 ppm
I20240502 14:14:39.496157 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:39.496690 17966 heartbeater.cc:347] Connected to a master server at 127.15.74.126:35925
I20240502 14:14:39.496819 17966 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:39.497193 17966 heartbeater.cc:511] Master 127.15.74.126:35925 requested a full tablet report, sending...
I20240502 14:14:39.498157 15657 webserver.cc:460] Webserver started at http://127.15.74.66:40213/ using document root <none> and password file <none>
I20240502 14:14:39.498209 17839 ts_manager.cc:194] Registered new tserver with Master: a897be9787a44b08aa7b722d260c94c6 (127.15.74.65:43309)
I20240502 14:14:39.498407 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:39.498489 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:39.498636 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:39.499287 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/ts-1-root/instance:
uuid: "f71a8633759944428cb0a90ed5887939"
format_stamp: "Formatted at 2024-05-02 14:14:39 on dist-test-slave-wbm6"
I20240502 14:14:39.499403 17839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54812
I20240502 14:14:39.501771 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:39.503252 17978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:39.503592 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 14:14:39.503706 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/ts-1-root
uuid: "f71a8633759944428cb0a90ed5887939"
format_stamp: "Formatted at 2024-05-02 14:14:39 on dist-test-slave-wbm6"
I20240502 14:14:39.503819 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:39.521342 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:39.521883 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:39.522379 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:39.523141 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:39.523211 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:39.523321 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:39.523377 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:39.535323 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.66:41409
I20240502 14:14:39.535370 18040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.66:41409 every 8 connection(s)
I20240502 14:14:39.536788 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:39.539779 18044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:39.540374 18045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:39.545079 18047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:39.545931 15657 server_base.cc:1034] running on GCE node
I20240502 14:14:39.546213 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:14:39.546277 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:14:39.546321 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659279546297 us; error 0 us; skew 500 ppm
I20240502 14:14:39.546483 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:39.547914 15657 webserver.cc:460] Webserver started at http://127.15.74.67:35273/ using document root <none> and password file <none>
I20240502 14:14:39.548184 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:39.548290 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:39.548499 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:39.549324 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/ts-2-root/instance:
uuid: "093ff5a1374d48948344215fd1d6c5f7"
format_stamp: "Formatted at 2024-05-02 14:14:39 on dist-test-slave-wbm6"
I20240502 14:14:39.550351 18041 heartbeater.cc:347] Connected to a master server at 127.15.74.126:35925
I20240502 14:14:39.550562 18041 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:39.550933 18041 heartbeater.cc:511] Master 127.15.74.126:35925 requested a full tablet report, sending...
I20240502 14:14:39.551846 17839 ts_manager.cc:194] Registered new tserver with Master: f71a8633759944428cb0a90ed5887939 (127.15.74.66:41409)
I20240502 14:14:39.552421 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:14:39.552770 17839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54824
I20240502 14:14:39.554060 18052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:39.554322 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:14:39.554430 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/ts-2-root
uuid: "093ff5a1374d48948344215fd1d6c5f7"
format_stamp: "Formatted at 2024-05-02 14:14:39 on dist-test-slave-wbm6"
I20240502 14:14:39.554548 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714659223241330-15657-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:39.589480 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:39.590078 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:39.590677 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:39.591516 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:39.591585 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:39.591687 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:39.591742 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:39.605297 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.67:35129
I20240502 14:14:39.605326 18114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.67:35129 every 8 connection(s)
I20240502 14:14:39.612282 18115 heartbeater.cc:347] Connected to a master server at 127.15.74.126:35925
I20240502 14:14:39.612426 18115 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:39.612773 18115 heartbeater.cc:511] Master 127.15.74.126:35925 requested a full tablet report, sending...
I20240502 14:14:39.613569 17839 ts_manager.cc:194] Registered new tserver with Master: 093ff5a1374d48948344215fd1d6c5f7 (127.15.74.67:35129)
I20240502 14:14:39.614090 15657 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007916359s
I20240502 14:14:39.614305 17839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54840
I20240502 14:14:40.500738 17966 heartbeater.cc:503] Master 127.15.74.126:35925 was elected leader, sending a full tablet report...
I20240502 14:14:40.554075 18041 heartbeater.cc:503] Master 127.15.74.126:35925 was elected leader, sending a full tablet report...
I20240502 14:14:40.615628 18115 heartbeater.cc:503] Master 127.15.74.126:35925 was elected leader, sending a full tablet report...
I20240502 14:14:41.374469 17892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:43.374897 17892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:45.375383 17892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:47.375933 17892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:49.376392 17892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:51.376899 17892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:51.698843 15657 tablet_server.cc:178] TabletServer@127.15.74.65:0 shutting down...
I20240502 14:14:51.713635 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:14:51.731678 15657 tablet_server.cc:194] TabletServer@127.15.74.65:0 shutdown complete.
I20240502 14:14:51.736086 15657 tablet_server.cc:178] TabletServer@127.15.74.66:0 shutting down...
I20240502 14:14:51.745693 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:14:51.758667 15657 tablet_server.cc:194] TabletServer@127.15.74.66:0 shutdown complete.
I20240502 14:14:51.761518 15657 tablet_server.cc:178] TabletServer@127.15.74.67:0 shutting down...
I20240502 14:14:51.770961 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:14:51.785308 15657 tablet_server.cc:194] TabletServer@127.15.74.67:0 shutdown complete.
I20240502 14:14:51.788681 15657 master.cc:537] Master@127.15.74.126:35925 shutting down...
I20240502 14:14:51.797591 15657 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:51.797866 15657 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.798344 15657 tablet_replica.cc:330] T 00000000000000000000000000000000 P bdd24413df134a4bab21c580d072fff2: stopping tablet replica
I20240502 14:14:51.813050 15657 master.cc:552] Master@127.15.74.126:35925 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12498 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240502 14:14:51.822445 15657 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.74.126:41795
I20240502 14:14:51.823016 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:51.825897 18119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:51.826995 18120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:51.827411 18122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:51.827581 15657 server_base.cc:1034] running on GCE node
I20240502 14:14:51.827992 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:14:51.828085 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:14:51.828147 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659291828125 us; error 0 us; skew 500 ppm
I20240502 14:14:51.828343 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:51.829732 15657 webserver.cc:460] Webserver started at http://127.15.74.126:46323/ using document root <none> and password file <none>
I20240502 14:14:51.829995 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:51.830086 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:51.830232 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:51.830919 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/master-0-root/instance:
uuid: "f4e919ffb1764e75999b631ca7bee5ba"
format_stamp: "Formatted at 2024-05-02 14:14:51 on dist-test-slave-wbm6"
I20240502 14:14:51.833614 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20240502 14:14:51.835140 18127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:51.835546 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 14:14:51.835662 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/master-0-root
uuid: "f4e919ffb1764e75999b631ca7bee5ba"
format_stamp: "Formatted at 2024-05-02 14:14:51 on dist-test-slave-wbm6"
I20240502 14:14:51.835790 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:51.856917 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:51.857635 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:51.871840 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.126:41795
I20240502 14:14:51.871873 18178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.126:41795 every 8 connection(s)
I20240502 14:14:51.873229 18179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.877554 18179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba: Bootstrap starting.
I20240502 14:14:51.878846 18179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.880589 18179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba: No bootstrap required, opened a new log
I20240502 14:14:51.881291 18179 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e919ffb1764e75999b631ca7bee5ba" member_type: VOTER }
I20240502 14:14:51.881434 18179 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.881489 18179 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e919ffb1764e75999b631ca7bee5ba, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.881668 18179 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e919ffb1764e75999b631ca7bee5ba" member_type: VOTER }
I20240502 14:14:51.881778 18179 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:14:51.881831 18179 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:14:51.881897 18179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.883733 18179 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e919ffb1764e75999b631ca7bee5ba" member_type: VOTER }
I20240502 14:14:51.884004 18179 leader_election.cc:302] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4e919ffb1764e75999b631ca7bee5ba; no voters:
I20240502 14:14:51.884465 18179 leader_election.cc:288] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:14:51.884591 18182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:51.885087 18182 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [term 1 LEADER]: Becoming Leader. State: Replica: f4e919ffb1764e75999b631ca7bee5ba, State: Running, Role: LEADER
I20240502 14:14:51.885378 18179 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:14:51.885344 18182 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e919ffb1764e75999b631ca7bee5ba" member_type: VOTER }
I20240502 14:14:51.886322 18184 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4e919ffb1764e75999b631ca7bee5ba. Latest consensus state: current_term: 1 leader_uuid: "f4e919ffb1764e75999b631ca7bee5ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e919ffb1764e75999b631ca7bee5ba" member_type: VOTER } }
I20240502 14:14:51.886322 18183 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4e919ffb1764e75999b631ca7bee5ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e919ffb1764e75999b631ca7bee5ba" member_type: VOTER } }
I20240502 14:14:51.886557 18184 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:51.886655 18183 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:51.887576 18187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:14:51.888991 18187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:14:51.891280 15657 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:14:51.892233 18187 catalog_manager.cc:1347] Generated new cluster ID: b4d0184a79a04d399f43211fa42ae6fd
I20240502 14:14:51.892338 18187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:14:51.908334 18187 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:14:51.909200 18187 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:14:51.915460 18187 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba: Generated new TSK 0
I20240502 14:14:51.915731 18187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:14:51.924568 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:51.927866 18200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:51.928498 18201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:51.929059 18203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:51.929279 15657 server_base.cc:1034] running on GCE node
I20240502 14:14:51.929590 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:14:51.929659 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:14:51.929697 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659291929681 us; error 0 us; skew 500 ppm
I20240502 14:14:51.929819 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:51.931090 15657 webserver.cc:460] Webserver started at http://127.15.74.65:46553/ using document root <none> and password file <none>
I20240502 14:14:51.931332 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:51.931406 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:51.931567 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:51.932196 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-0-root/instance:
uuid: "cee40f1e534f4fafbfcc8d7dca1c2729"
format_stamp: "Formatted at 2024-05-02 14:14:51 on dist-test-slave-wbm6"
I20240502 14:14:51.934825 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:51.936528 18208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:51.936971 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 14:14:51.937076 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-0-root
uuid: "cee40f1e534f4fafbfcc8d7dca1c2729"
format_stamp: "Formatted at 2024-05-02 14:14:51 on dist-test-slave-wbm6"
I20240502 14:14:51.937213 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:51.964205 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:51.964886 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:51.965543 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:51.966534 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:51.966606 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:51.966702 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:51.966758 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:51.981506 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.65:37245
I20240502 14:14:51.981545 18270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.65:37245 every 8 connection(s)
I20240502 14:14:51.983520 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:51.987783 18276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:51.990804 18275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:51.991533 18278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:51.992142 15657 server_base.cc:1034] running on GCE node
I20240502 14:14:51.992486 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:14:51.992594 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:14:51.992652 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659291992627 us; error 0 us; skew 500 ppm
I20240502 14:14:51.992870 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:51.993468 18271 heartbeater.cc:347] Connected to a master server at 127.15.74.126:41795
I20240502 14:14:51.993628 18271 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:51.994127 18271 heartbeater.cc:511] Master 127.15.74.126:41795 requested a full tablet report, sending...
I20240502 14:14:51.994347 15657 webserver.cc:460] Webserver started at http://127.15.74.66:42429/ using document root <none> and password file <none>
I20240502 14:14:51.994619 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:51.994735 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:51.994877 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:51.995095 18144 ts_manager.cc:194] Registered new tserver with Master: cee40f1e534f4fafbfcc8d7dca1c2729 (127.15.74.65:37245)
I20240502 14:14:51.995584 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-1-root/instance:
uuid: "d5a6416b63234bb8857483476a6071dd"
format_stamp: "Formatted at 2024-05-02 14:14:51 on dist-test-slave-wbm6"
I20240502 14:14:51.996263 18144 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53936
I20240502 14:14:51.998142 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:51.999970 18283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:52.000417 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 14:14:52.000524 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-1-root
uuid: "d5a6416b63234bb8857483476a6071dd"
format_stamp: "Formatted at 2024-05-02 14:14:51 on dist-test-slave-wbm6"
I20240502 14:14:52.000636 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:52.012276 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:52.012857 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:52.013383 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:52.014174 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:52.014248 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:52.014319 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:52.014365 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:52.027118 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.66:43325
I20240502 14:14:52.027141 18345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.66:43325 every 8 connection(s)
I20240502 14:14:52.028940 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:52.031910 18349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:52.032655 18350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:52.033288 18352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:52.034436 15657 server_base.cc:1034] running on GCE node
I20240502 14:14:52.034777 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:14:52.034875 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:14:52.034976 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659292034949 us; error 0 us; skew 500 ppm
I20240502 14:14:52.035189 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:52.037050 15657 webserver.cc:460] Webserver started at http://127.15.74.67:42575/ using document root <none> and password file <none>
I20240502 14:14:52.037374 18346 heartbeater.cc:347] Connected to a master server at 127.15.74.126:41795
I20240502 14:14:52.037400 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:52.037582 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:52.037636 18346 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:52.037744 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:52.038161 18346 heartbeater.cc:511] Master 127.15.74.126:41795 requested a full tablet report, sending...
I20240502 14:14:52.038653 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-2-root/instance:
uuid: "5869265d70974c839d0db1ad00ffd82d"
format_stamp: "Formatted at 2024-05-02 14:14:52 on dist-test-slave-wbm6"
I20240502 14:14:52.039245 18144 ts_manager.cc:194] Registered new tserver with Master: d5a6416b63234bb8857483476a6071dd (127.15.74.66:43325)
I20240502 14:14:52.040088 18144 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53944
I20240502 14:14:52.041855 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 14:14:52.043565 18357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:52.043942 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240502 14:14:52.044063 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-2-root
uuid: "5869265d70974c839d0db1ad00ffd82d"
format_stamp: "Formatted at 2024-05-02 14:14:52 on dist-test-slave-wbm6"
I20240502 14:14:52.044183 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:52.063771 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:52.064527 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:52.065218 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:52.077571 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:52.077664 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:52.077745 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:52.077796 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:52.089864 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.67:45141
I20240502 14:14:52.089876 18422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.67:45141 every 8 connection(s)
I20240502 14:14:52.091400 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:52.094175 18426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:52.095014 18427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:52.095230 15657 server_base.cc:1034] running on GCE node
W20240502 14:14:52.095796 18429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:52.096166 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:14:52.096237 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:14:52.096272 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659292096268 us; error 0 us; skew 500 ppm
I20240502 14:14:52.096428 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:52.097623 18423 heartbeater.cc:347] Connected to a master server at 127.15.74.126:41795
I20240502 14:14:52.097733 18423 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:52.097965 15657 webserver.cc:460] Webserver started at http://127.15.74.68:41281/ using document root <none> and password file <none>
I20240502 14:14:52.098080 18423 heartbeater.cc:511] Master 127.15.74.126:41795 requested a full tablet report, sending...
I20240502 14:14:52.098186 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:52.098265 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:52.098383 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:52.098989 18144 ts_manager.cc:194] Registered new tserver with Master: 5869265d70974c839d0db1ad00ffd82d (127.15.74.67:45141)
I20240502 14:14:52.099151 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-3-root/instance:
uuid: "236522fd377c4bf8823e9aa04fb4d728"
format_stamp: "Formatted at 2024-05-02 14:14:52 on dist-test-slave-wbm6"
I20240502 14:14:52.100075 18144 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53956
I20240502 14:14:52.102067 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:52.103562 18434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:52.103945 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 14:14:52.104035 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-3-root
uuid: "236522fd377c4bf8823e9aa04fb4d728"
format_stamp: "Formatted at 2024-05-02 14:14:52 on dist-test-slave-wbm6"
I20240502 14:14:52.104125 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714659223241330-15657-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:52.117669 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:52.118192 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:52.118702 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:52.119596 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:52.119653 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:52.119711 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:52.119740 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:52.130604 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.68:35483
I20240502 14:14:52.130638 18496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.68:35483 every 8 connection(s)
I20240502 14:14:52.137416 18497 heartbeater.cc:347] Connected to a master server at 127.15.74.126:41795
I20240502 14:14:52.137593 18497 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:52.137995 18497 heartbeater.cc:511] Master 127.15.74.126:41795 requested a full tablet report, sending...
I20240502 14:14:52.138840 18144 ts_manager.cc:194] Registered new tserver with Master: 236522fd377c4bf8823e9aa04fb4d728 (127.15.74.68:35483)
I20240502 14:14:52.139578 15657 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.007940885s
I20240502 14:14:52.139715 18144 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53964
I20240502 14:14:52.997582 18271 heartbeater.cc:503] Master 127.15.74.126:41795 was elected leader, sending a full tablet report...
I20240502 14:14:53.041867 18346 heartbeater.cc:503] Master 127.15.74.126:41795 was elected leader, sending a full tablet report...
I20240502 14:14:53.101343 18423 heartbeater.cc:503] Master 127.15.74.126:41795 was elected leader, sending a full tablet report...
I20240502 14:14:53.141184 18497 heartbeater.cc:503] Master 127.15.74.126:41795 was elected leader, sending a full tablet report...
I20240502 14:14:53.174733 15657 test_util.cc:268] Using random seed: 974429112
I20240502 14:14:53.186821 18144 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38492:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 14:14:53.204106 18388 tablet_service.cc:1467] Processing CreateTablet for tablet eb0cd0c49bd74403a2f2eca6a2188823 (DEFAULT_TABLE table=test-workload [id=1fda6b6375b44e12abd6b2e841dd7654]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.204665 18388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0cd0c49bd74403a2f2eca6a2188823. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.206735 18311 tablet_service.cc:1467] Processing CreateTablet for tablet eb0cd0c49bd74403a2f2eca6a2188823 (DEFAULT_TABLE table=test-workload [id=1fda6b6375b44e12abd6b2e841dd7654]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.207165 18311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0cd0c49bd74403a2f2eca6a2188823. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.212468 18518 tablet_bootstrap.cc:492] T eb0cd0c49bd74403a2f2eca6a2188823 P 5869265d70974c839d0db1ad00ffd82d: Bootstrap starting.
I20240502 14:14:53.214689 18518 tablet_bootstrap.cc:654] T eb0cd0c49bd74403a2f2eca6a2188823 P 5869265d70974c839d0db1ad00ffd82d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.215658 18236 tablet_service.cc:1467] Processing CreateTablet for tablet eb0cd0c49bd74403a2f2eca6a2188823 (DEFAULT_TABLE table=test-workload [id=1fda6b6375b44e12abd6b2e841dd7654]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.215847 18519 tablet_bootstrap.cc:492] T eb0cd0c49bd74403a2f2eca6a2188823 P d5a6416b63234bb8857483476a6071dd: Bootstrap starting.
I20240502 14:14:53.216238 18236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0cd0c49bd74403a2f2eca6a2188823. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.217271 18518 tablet_bootstrap.cc:492] T eb0cd0c49bd74403a2f2eca6a2188823 P 5869265d70974c839d0db1ad00ffd82d: No bootstrap required, opened a new log
I20240502 14:14:53.217401 18518 ts_tablet_manager.cc:1367] T eb0cd0c49bd74403a2f2eca6a2188823 P 5869265d70974c839d0db1ad00ffd82d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240502 14:14:53.218070 18519 tablet_bootstrap.cc:654] T eb0cd0c49bd74403a2f2eca6a2188823 P d5a6416b63234bb8857483476a6071dd: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.218156 18518 raft_consensus.cc:348] T eb0cd0c49bd74403a2f2eca6a2188823 P 5869265d70974c839d0db1ad00ffd82d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5869265d70974c839d0db1ad00ffd82d" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45141 } } peers { permanent_uuid: "cee40f1e534f4fafbfcc8d7dca1c2729" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37245 } } peers { permanent_uuid: "d5a6416b63234bb8857483476a6071dd" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43325 } }
I20240502 14:14:53.218356 18518 raft_consensus.cc:374] T eb0cd0c49bd74403a2f2eca6a2188823 P 5869265d70974c839d0db1ad00ffd82d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.218459 18518 raft_consensus.cc:729] T eb0cd0c49bd74403a2f2eca6a2188823 P 5869265d70974c839d0db1ad00ffd82d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5869265d70974c839d0db1ad00ffd82d, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.218719 18518 consensus_queue.cc:260] T eb0cd0c49bd74403a2f2eca6a2188823 P 5869265d70974c839d0db1ad00ffd82d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5869265d70974c839d0db1ad00ffd82d" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45141 } } peers { permanent_uuid: "cee40f1e534f4fafbfcc8d7dca1c2729" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37245 } } peers { permanent_uuid: "d5a6416b63234bb8857483476a6071dd" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43325 } }
I20240502 14:14:53.219846 18519 tablet_bootstrap.cc:492] T eb0cd0c49bd74403a2f2eca6a2188823 P d5a6416b63234bb8857483476a6071dd: No bootstrap required, opened a new log
I20240502 14:14:53.219975 18519 ts_tablet_manager.cc:1367] T eb0cd0c49bd74403a2f2eca6a2188823 P d5a6416b63234bb8857483476a6071dd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:53.220405 18518 ts_tablet_manager.cc:1398] T eb0cd0c49bd74403a2f2eca6a2188823 P 5869265d70974c839d0db1ad00ffd82d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 14:14:53.220929 18519 raft_consensus.cc:348] T eb0cd0c49bd74403a2f2eca6a2188823 P d5a6416b63234bb8857483476a6071dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5869265d70974c839d0db1ad00ffd82d" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45141 } } peers { permanent_uuid: "cee40f1e534f4fafbfcc8d7dca1c2729" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37245 } } peers { permanent_uuid: "d5a6416b63234bb8857483476a6071dd" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43325 } }
I20240502 14:14:53.221166 18519 raft_consensus.cc:374] T eb0cd0c49bd74403a2f2eca6a2188823 P d5a6416b63234bb8857483476a6071dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.221271 18519 raft_consensus.cc:729] T eb0cd0c49bd74403a2f2eca6a2188823 P d5a6416b63234bb8857483476a6071dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5a6416b63234bb8857483476a6071dd, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.221498 18519 consensus_queue.cc:260] T eb0cd0c49bd74403a2f2eca6a2188823 P d5a6416b63234bb8857483476a6071dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5869265d70974c839d0db1ad00ffd82d" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45141 } } peers { permanent_uuid: "cee40f1e534f4fafbfcc8d7dca1c2729" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37245 } } peers { permanent_uuid: "d5a6416b63234bb8857483476a6071dd" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43325 } }
I20240502 14:14:53.222424 18519 ts_tablet_manager.cc:1398] T eb0cd0c49bd74403a2f2eca6a2188823 P d5a6416b63234bb8857483476a6071dd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:53.222820 18524 tablet_bootstrap.cc:492] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729: Bootstrap starting.
I20240502 14:14:53.224454 18524 tablet_bootstrap.cc:654] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.226341 18524 tablet_bootstrap.cc:492] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729: No bootstrap required, opened a new log
I20240502 14:14:53.226454 18524 ts_tablet_manager.cc:1367] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:53.227054 18524 raft_consensus.cc:348] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5869265d70974c839d0db1ad00ffd82d" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45141 } } peers { permanent_uuid: "cee40f1e534f4fafbfcc8d7dca1c2729" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37245 } } peers { permanent_uuid: "d5a6416b63234bb8857483476a6071dd" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43325 } }
I20240502 14:14:53.227217 18524 raft_consensus.cc:374] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.227267 18524 raft_consensus.cc:729] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee40f1e534f4fafbfcc8d7dca1c2729, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.227491 18524 consensus_queue.cc:260] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5869265d70974c839d0db1ad00ffd82d" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45141 } } peers { permanent_uuid: "cee40f1e534f4fafbfcc8d7dca1c2729" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37245 } } peers { permanent_uuid: "d5a6416b63234bb8857483476a6071dd" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43325 } }
I20240502 14:14:53.228062 18524 ts_tablet_manager.cc:1398] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240502 14:14:53.233719 18272 tablet.cc:2369] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:53.410373 18526 raft_consensus.cc:482] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:53.410599 18526 raft_consensus.cc:504] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5869265d70974c839d0db1ad00ffd82d" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45141 } } peers { permanent_uuid: "cee40f1e534f4fafbfcc8d7dca1c2729" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37245 } } peers { permanent_uuid: "d5a6416b63234bb8857483476a6071dd" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43325 } }
I20240502 14:14:53.411542 18526 leader_election.cc:288] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5869265d70974c839d0db1ad00ffd82d (127.15.74.67:45141), d5a6416b63234bb8857483476a6071dd (127.15.74.66:43325)
I20240502 14:14:53.417635 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0cd0c49bd74403a2f2eca6a2188823" candidate_uuid: "cee40f1e534f4fafbfcc8d7dca1c2729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5869265d70974c839d0db1ad00ffd82d" is_pre_election: true
I20240502 14:14:53.417902 18398 raft_consensus.cc:2454] T eb0cd0c49bd74403a2f2eca6a2188823 P 5869265d70974c839d0db1ad00ffd82d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee40f1e534f4fafbfcc8d7dca1c2729 in term 0.
I20240502 14:14:53.417946 18321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0cd0c49bd74403a2f2eca6a2188823" candidate_uuid: "cee40f1e534f4fafbfcc8d7dca1c2729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5a6416b63234bb8857483476a6071dd" is_pre_election: true
I20240502 14:14:53.418148 18321 raft_consensus.cc:2454] T eb0cd0c49bd74403a2f2eca6a2188823 P d5a6416b63234bb8857483476a6071dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee40f1e534f4fafbfcc8d7dca1c2729 in term 0.
I20240502 14:14:53.418341 18212 leader_election.cc:302] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5869265d70974c839d0db1ad00ffd82d, cee40f1e534f4fafbfcc8d7dca1c2729; no voters:
I20240502 14:14:53.418576 18526 raft_consensus.cc:2781] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:53.418658 18526 raft_consensus.cc:482] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:53.418751 18526 raft_consensus.cc:3037] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.420776 18526 raft_consensus.cc:504] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5869265d70974c839d0db1ad00ffd82d" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45141 } } peers { permanent_uuid: "cee40f1e534f4fafbfcc8d7dca1c2729" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37245 } } peers { permanent_uuid: "d5a6416b63234bb8857483476a6071dd" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43325 } }
I20240502 14:14:53.421289 18526 leader_election.cc:288] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [CANDIDATE]: Term 1 election: Requested vote from peers 5869265d70974c839d0db1ad00ffd82d (127.15.74.67:45141), d5a6416b63234bb8857483476a6071dd (127.15.74.66:43325)
I20240502 14:14:53.421764 18321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0cd0c49bd74403a2f2eca6a2188823" candidate_uuid: "cee40f1e534f4fafbfcc8d7dca1c2729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5a6416b63234bb8857483476a6071dd"
I20240502 14:14:53.421764 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0cd0c49bd74403a2f2eca6a2188823" candidate_uuid: "cee40f1e534f4fafbfcc8d7dca1c2729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5869265d70974c839d0db1ad00ffd82d"
I20240502 14:14:53.421972 18398 raft_consensus.cc:3037] T eb0cd0c49bd74403a2f2eca6a2188823 P 5869265d70974c839d0db1ad00ffd82d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.421972 18321 raft_consensus.cc:3037] T eb0cd0c49bd74403a2f2eca6a2188823 P d5a6416b63234bb8857483476a6071dd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.424737 18398 raft_consensus.cc:2454] T eb0cd0c49bd74403a2f2eca6a2188823 P 5869265d70974c839d0db1ad00ffd82d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee40f1e534f4fafbfcc8d7dca1c2729 in term 1.
I20240502 14:14:53.424737 18321 raft_consensus.cc:2454] T eb0cd0c49bd74403a2f2eca6a2188823 P d5a6416b63234bb8857483476a6071dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee40f1e534f4fafbfcc8d7dca1c2729 in term 1.
I20240502 14:14:53.425141 18212 leader_election.cc:302] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5869265d70974c839d0db1ad00ffd82d, cee40f1e534f4fafbfcc8d7dca1c2729; no voters:
I20240502 14:14:53.425348 18526 raft_consensus.cc:2781] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:53.425763 18526 raft_consensus.cc:686] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [term 1 LEADER]: Becoming Leader. State: Replica: cee40f1e534f4fafbfcc8d7dca1c2729, State: Running, Role: LEADER
I20240502 14:14:53.425993 18526 consensus_queue.cc:237] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5869265d70974c839d0db1ad00ffd82d" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45141 } } peers { permanent_uuid: "cee40f1e534f4fafbfcc8d7dca1c2729" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37245 } } peers { permanent_uuid: "d5a6416b63234bb8857483476a6071dd" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43325 } }
I20240502 14:14:53.428532 18144 catalog_manager.cc:5484] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 reported cstate change: term changed from 0 to 1, leader changed from <none> to cee40f1e534f4fafbfcc8d7dca1c2729 (127.15.74.65). New cstate: current_term: 1 leader_uuid: "cee40f1e534f4fafbfcc8d7dca1c2729" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5869265d70974c839d0db1ad00ffd82d" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cee40f1e534f4fafbfcc8d7dca1c2729" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 37245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5a6416b63234bb8857483476a6071dd" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43325 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:53.885628 18526 consensus_queue.cc:1035] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5869265d70974c839d0db1ad00ffd82d" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.891638 18197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:53.892661 18197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:53.892760 18197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:53.892839 18529 consensus_queue.cc:1035] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5a6416b63234bb8857483476a6071dd" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 43325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:55.893179 18197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:55.893643 18197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:55.893734 18197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:57.894155 18197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:57.894623 18197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:57.894712 18197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:14:59.895114 18197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:14:59.895668 18197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:14:59.895792 18197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:01.896212 18197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:01.896741 18197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:15:01.896836 18197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:03.897348 18197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:03.897877 18197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:15:03.897966 18197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:04.491638 15657 tablet_server.cc:178] TabletServer@127.15.74.65:0 shutting down...
I20240502 14:15:04.503050 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:15:04.503386 15657 tablet_replica.cc:330] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729: stopping tablet replica
I20240502 14:15:04.503600 15657 raft_consensus.cc:2229] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:04.503983 15657 raft_consensus.cc:2258] T eb0cd0c49bd74403a2f2eca6a2188823 P cee40f1e534f4fafbfcc8d7dca1c2729 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:04.518147 15657 tablet_server.cc:194] TabletServer@127.15.74.65:0 shutdown complete.
I20240502 14:15:04.521256 15657 tablet_server.cc:178] TabletServer@127.15.74.66:0 shutting down...
I20240502 14:15:04.529590 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:15:04.529865 15657 tablet_replica.cc:330] T eb0cd0c49bd74403a2f2eca6a2188823 P d5a6416b63234bb8857483476a6071dd: stopping tablet replica
I20240502 14:15:04.530035 15657 raft_consensus.cc:2229] T eb0cd0c49bd74403a2f2eca6a2188823 P d5a6416b63234bb8857483476a6071dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:04.530225 15657 raft_consensus.cc:2258] T eb0cd0c49bd74403a2f2eca6a2188823 P d5a6416b63234bb8857483476a6071dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:04.544044 15657 tablet_server.cc:194] TabletServer@127.15.74.66:0 shutdown complete.
I20240502 14:15:04.547302 15657 tablet_server.cc:178] TabletServer@127.15.74.67:0 shutting down...
I20240502 14:15:04.555335 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:15:04.555624 15657 tablet_replica.cc:330] T eb0cd0c49bd74403a2f2eca6a2188823 P 5869265d70974c839d0db1ad00ffd82d: stopping tablet replica
I20240502 14:15:04.555815 15657 raft_consensus.cc:2229] T eb0cd0c49bd74403a2f2eca6a2188823 P 5869265d70974c839d0db1ad00ffd82d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:04.555992 15657 raft_consensus.cc:2258] T eb0cd0c49bd74403a2f2eca6a2188823 P 5869265d70974c839d0db1ad00ffd82d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:04.569468 15657 tablet_server.cc:194] TabletServer@127.15.74.67:0 shutdown complete.
I20240502 14:15:04.572407 15657 tablet_server.cc:178] TabletServer@127.15.74.68:0 shutting down...
I20240502 14:15:04.579819 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:15:04.592481 15657 tablet_server.cc:194] TabletServer@127.15.74.68:0 shutdown complete.
I20240502 14:15:04.606878 15657 master.cc:537] Master@127.15.74.126:41795 shutting down...
I20240502 14:15:04.613898 15657 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:04.614137 15657 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:04.614217 15657 tablet_replica.cc:330] T 00000000000000000000000000000000 P f4e919ffb1764e75999b631ca7bee5ba: stopping tablet replica
I20240502 14:15:04.627563 15657 master.cc:552] Master@127.15.74.126:41795 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12816 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240502 14:15:04.639055 15657 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.74.126:46613
I20240502 14:15:04.639572 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:04.641902 18543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:04.642524 18544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:04.642753 18546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:04.643415 15657 server_base.cc:1034] running on GCE node
I20240502 14:15:04.643669 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:15:04.643720 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:15:04.643757 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659304643752 us; error 0 us; skew 500 ppm
I20240502 14:15:04.643914 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:04.645215 15657 webserver.cc:460] Webserver started at http://127.15.74.126:43871/ using document root <none> and password file <none>
I20240502 14:15:04.645426 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:04.645480 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:04.645583 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:04.646260 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/master-0-root/instance:
uuid: "b1c2261244d043229835a06bb3127c57"
format_stamp: "Formatted at 2024-05-02 14:15:04 on dist-test-slave-wbm6"
I20240502 14:15:04.648898 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:04.650532 18551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:04.650928 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:15:04.651014 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/master-0-root
uuid: "b1c2261244d043229835a06bb3127c57"
format_stamp: "Formatted at 2024-05-02 14:15:04 on dist-test-slave-wbm6"
I20240502 14:15:04.651110 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:04.658783 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:04.659171 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:04.669075 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.126:46613
I20240502 14:15:04.669095 18602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.126:46613 every 8 connection(s)
I20240502 14:15:04.670387 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.674470 18603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57: Bootstrap starting.
I20240502 14:15:04.675626 18603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.677045 18603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57: No bootstrap required, opened a new log
I20240502 14:15:04.677595 18603 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c2261244d043229835a06bb3127c57" member_type: VOTER }
I20240502 14:15:04.677709 18603 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.677759 18603 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1c2261244d043229835a06bb3127c57, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.677942 18603 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c2261244d043229835a06bb3127c57" member_type: VOTER }
I20240502 14:15:04.678045 18603 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:15:04.678100 18603 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:15:04.678164 18603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.679891 18603 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c2261244d043229835a06bb3127c57" member_type: VOTER }
I20240502 14:15:04.680061 18603 leader_election.cc:302] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1c2261244d043229835a06bb3127c57; no voters:
I20240502 14:15:04.680406 18603 leader_election.cc:288] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:15:04.680534 18606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:04.681005 18606 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [term 1 LEADER]: Becoming Leader. State: Replica: b1c2261244d043229835a06bb3127c57, State: Running, Role: LEADER
I20240502 14:15:04.681262 18603 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:15:04.681219 18606 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c2261244d043229835a06bb3127c57" member_type: VOTER }
I20240502 14:15:04.682039 18607 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1c2261244d043229835a06bb3127c57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c2261244d043229835a06bb3127c57" member_type: VOTER } }
I20240502 14:15:04.681988 18608 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1c2261244d043229835a06bb3127c57. Latest consensus state: current_term: 1 leader_uuid: "b1c2261244d043229835a06bb3127c57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c2261244d043229835a06bb3127c57" member_type: VOTER } }
I20240502 14:15:04.682200 18607 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [sys.catalog]: This master's current role is: LEADER
I20240502 14:15:04.682288 18608 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [sys.catalog]: This master's current role is: LEADER
I20240502 14:15:04.683706 18611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:15:04.685056 18611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:15:04.686501 15657 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:15:04.687893 18611 catalog_manager.cc:1347] Generated new cluster ID: cf6fa1118960477b912a89d78cc22b60
I20240502 14:15:04.687979 18611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:15:04.697515 18611 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:15:04.698231 18611 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:15:04.704681 18611 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57: Generated new TSK 0
I20240502 14:15:04.704871 18611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:15:04.719151 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:04.721722 18624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:04.722214 18625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:04.722787 18627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:04.722992 15657 server_base.cc:1034] running on GCE node
I20240502 14:15:04.723189 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:15:04.723237 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:15:04.723270 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659304723265 us; error 0 us; skew 500 ppm
I20240502 14:15:04.723443 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:04.724542 15657 webserver.cc:460] Webserver started at http://127.15.74.65:36145/ using document root <none> and password file <none>
I20240502 14:15:04.724743 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:04.724794 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:04.724906 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:04.725554 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/ts-0-root/instance:
uuid: "1708653910d94234bd9fd7865428025c"
format_stamp: "Formatted at 2024-05-02 14:15:04 on dist-test-slave-wbm6"
I20240502 14:15:04.727900 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20240502 14:15:04.729478 18632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:04.729841 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 14:15:04.729933 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/ts-0-root
uuid: "1708653910d94234bd9fd7865428025c"
format_stamp: "Formatted at 2024-05-02 14:15:04 on dist-test-slave-wbm6"
I20240502 14:15:04.730017 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:04.737426 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:04.737831 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:04.738268 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:04.739068 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:04.739115 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:04.739169 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:04.739195 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:04.750452 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.65:36593
I20240502 14:15:04.750476 18694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.65:36593 every 8 connection(s)
I20240502 14:15:04.752049 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:04.754796 18699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:04.755466 18700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:04.755561 18702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:04.756529 15657 server_base.cc:1034] running on GCE node
I20240502 14:15:04.756750 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:15:04.756803 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:15:04.756856 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659304756852 us; error 0 us; skew 500 ppm
I20240502 14:15:04.756980 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:04.758441 15657 webserver.cc:460] Webserver started at http://127.15.74.66:35227/ using document root <none> and password file <none>
I20240502 14:15:04.758664 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:04.758736 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:04.758844 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:04.759307 18695 heartbeater.cc:347] Connected to a master server at 127.15.74.126:46613
I20240502 14:15:04.759467 18695 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:04.759622 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/ts-1-root/instance:
uuid: "d9f992bf75594e43b909300b21f59327"
format_stamp: "Formatted at 2024-05-02 14:15:04 on dist-test-slave-wbm6"
I20240502 14:15:04.759835 18695 heartbeater.cc:511] Master 127.15.74.126:46613 requested a full tablet report, sending...
I20240502 14:15:04.760758 18568 ts_manager.cc:194] Registered new tserver with Master: 1708653910d94234bd9fd7865428025c (127.15.74.65:36593)
I20240502 14:15:04.762225 18568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40018
I20240502 14:15:04.762580 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:15:04.764324 18707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:04.764611 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 14:15:04.764701 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/ts-1-root
uuid: "d9f992bf75594e43b909300b21f59327"
format_stamp: "Formatted at 2024-05-02 14:15:04 on dist-test-slave-wbm6"
I20240502 14:15:04.764792 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:04.786864 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:04.787360 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:04.787920 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:04.788722 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:04.788775 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:04.788846 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:04.788878 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:04.799551 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.66:36963
I20240502 14:15:04.799585 18769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.66:36963 every 8 connection(s)
I20240502 14:15:04.801007 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:04.803658 18773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:04.804252 18774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:04.805115 15657 server_base.cc:1034] running on GCE node
W20240502 14:15:04.805524 18776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:04.805887 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:15:04.805969 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:15:04.806072 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659304806064 us; error 0 us; skew 500 ppm
I20240502 14:15:04.806250 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:04.807665 15657 webserver.cc:460] Webserver started at http://127.15.74.67:36871/ using document root <none> and password file <none>
I20240502 14:15:04.807921 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:04.808002 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:04.808116 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:04.808157 18770 heartbeater.cc:347] Connected to a master server at 127.15.74.126:46613
I20240502 14:15:04.808275 18770 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:04.808647 18770 heartbeater.cc:511] Master 127.15.74.126:46613 requested a full tablet report, sending...
I20240502 14:15:04.808924 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/ts-2-root/instance:
uuid: "890d461ec94647c88541b1165deee654"
format_stamp: "Formatted at 2024-05-02 14:15:04 on dist-test-slave-wbm6"
I20240502 14:15:04.809561 18568 ts_manager.cc:194] Registered new tserver with Master: d9f992bf75594e43b909300b21f59327 (127.15.74.66:36963)
I20240502 14:15:04.810591 18568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40020
I20240502 14:15:04.811954 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240502 14:15:04.813526 18781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:04.813905 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 14:15:04.814000 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/ts-2-root
uuid: "890d461ec94647c88541b1165deee654"
format_stamp: "Formatted at 2024-05-02 14:15:04 on dist-test-slave-wbm6"
I20240502 14:15:04.814110 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714659223241330-15657-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:04.842758 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:04.843333 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:04.843931 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:04.844771 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:04.844827 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:04.844889 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:04.844930 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:04.856648 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.67:39651
I20240502 14:15:04.856676 18843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.67:39651 every 8 connection(s)
I20240502 14:15:04.863692 18844 heartbeater.cc:347] Connected to a master server at 127.15.74.126:46613
I20240502 14:15:04.863847 18844 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:04.864243 18844 heartbeater.cc:511] Master 127.15.74.126:46613 requested a full tablet report, sending...
I20240502 14:15:04.865185 18568 ts_manager.cc:194] Registered new tserver with Master: 890d461ec94647c88541b1165deee654 (127.15.74.67:39651)
I20240502 14:15:04.865535 15657 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007991132s
I20240502 14:15:04.866030 18568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40022
I20240502 14:15:05.763674 18695 heartbeater.cc:503] Master 127.15.74.126:46613 was elected leader, sending a full tablet report...
I20240502 14:15:05.812017 18770 heartbeater.cc:503] Master 127.15.74.126:46613 was elected leader, sending a full tablet report...
I20240502 14:15:05.867537 18844 heartbeater.cc:503] Master 127.15.74.126:46613 was elected leader, sending a full tablet report...
I20240502 14:15:05.901175 15657 test_util.cc:268] Using random seed: 987155569
I20240502 14:15:05.913174 18568 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40032:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:15:05.913774 18568 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:15:05.932186 18660 tablet_service.cc:1467] Processing CreateTablet for tablet a8db8a87a56b4227941f10ccd88fc0fe (DEFAULT_TABLE table=test-workload [id=6e453e42975e431790a8206f1f1be743]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.932716 18660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8db8a87a56b4227941f10ccd88fc0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.933944 18809 tablet_service.cc:1467] Processing CreateTablet for tablet a8db8a87a56b4227941f10ccd88fc0fe (DEFAULT_TABLE table=test-workload [id=6e453e42975e431790a8206f1f1be743]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.934365 18809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8db8a87a56b4227941f10ccd88fc0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.937429 18735 tablet_service.cc:1467] Processing CreateTablet for tablet a8db8a87a56b4227941f10ccd88fc0fe (DEFAULT_TABLE table=test-workload [id=6e453e42975e431790a8206f1f1be743]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.937852 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8db8a87a56b4227941f10ccd88fc0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.940335 18865 tablet_bootstrap.cc:492] T a8db8a87a56b4227941f10ccd88fc0fe P 1708653910d94234bd9fd7865428025c: Bootstrap starting.
I20240502 14:15:05.941668 18865 tablet_bootstrap.cc:654] T a8db8a87a56b4227941f10ccd88fc0fe P 1708653910d94234bd9fd7865428025c: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.942373 18866 tablet_bootstrap.cc:492] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654: Bootstrap starting.
I20240502 14:15:05.944108 18865 tablet_bootstrap.cc:492] T a8db8a87a56b4227941f10ccd88fc0fe P 1708653910d94234bd9fd7865428025c: No bootstrap required, opened a new log
I20240502 14:15:05.944187 18866 tablet_bootstrap.cc:654] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.944280 18865 ts_tablet_manager.cc:1367] T a8db8a87a56b4227941f10ccd88fc0fe P 1708653910d94234bd9fd7865428025c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:15:05.944929 18868 tablet_bootstrap.cc:492] T a8db8a87a56b4227941f10ccd88fc0fe P d9f992bf75594e43b909300b21f59327: Bootstrap starting.
I20240502 14:15:05.945231 18865 raft_consensus.cc:348] T a8db8a87a56b4227941f10ccd88fc0fe P 1708653910d94234bd9fd7865428025c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1708653910d94234bd9fd7865428025c" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 36593 } } peers { permanent_uuid: "890d461ec94647c88541b1165deee654" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39651 } } peers { permanent_uuid: "d9f992bf75594e43b909300b21f59327" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 36963 } }
I20240502 14:15:05.945539 18865 raft_consensus.cc:374] T a8db8a87a56b4227941f10ccd88fc0fe P 1708653910d94234bd9fd7865428025c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.945706 18865 raft_consensus.cc:729] T a8db8a87a56b4227941f10ccd88fc0fe P 1708653910d94234bd9fd7865428025c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1708653910d94234bd9fd7865428025c, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.946045 18865 consensus_queue.cc:260] T a8db8a87a56b4227941f10ccd88fc0fe P 1708653910d94234bd9fd7865428025c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1708653910d94234bd9fd7865428025c" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 36593 } } peers { permanent_uuid: "890d461ec94647c88541b1165deee654" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39651 } } peers { permanent_uuid: "d9f992bf75594e43b909300b21f59327" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 36963 } }
I20240502 14:15:05.946372 18868 tablet_bootstrap.cc:654] T a8db8a87a56b4227941f10ccd88fc0fe P d9f992bf75594e43b909300b21f59327: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.947078 18865 ts_tablet_manager.cc:1398] T a8db8a87a56b4227941f10ccd88fc0fe P 1708653910d94234bd9fd7865428025c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:05.948601 18866 tablet_bootstrap.cc:492] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654: No bootstrap required, opened a new log
I20240502 14:15:05.949509 18866 ts_tablet_manager.cc:1367] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20240502 14:15:05.949970 18868 tablet_bootstrap.cc:492] T a8db8a87a56b4227941f10ccd88fc0fe P d9f992bf75594e43b909300b21f59327: No bootstrap required, opened a new log
I20240502 14:15:05.950122 18868 ts_tablet_manager.cc:1367] T a8db8a87a56b4227941f10ccd88fc0fe P d9f992bf75594e43b909300b21f59327: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:15:05.950438 18866 raft_consensus.cc:348] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1708653910d94234bd9fd7865428025c" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 36593 } } peers { permanent_uuid: "890d461ec94647c88541b1165deee654" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39651 } } peers { permanent_uuid: "d9f992bf75594e43b909300b21f59327" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 36963 } }
I20240502 14:15:05.950654 18866 raft_consensus.cc:374] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.950807 18866 raft_consensus.cc:729] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890d461ec94647c88541b1165deee654, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.950909 18868 raft_consensus.cc:348] T a8db8a87a56b4227941f10ccd88fc0fe P d9f992bf75594e43b909300b21f59327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1708653910d94234bd9fd7865428025c" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 36593 } } peers { permanent_uuid: "890d461ec94647c88541b1165deee654" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39651 } } peers { permanent_uuid: "d9f992bf75594e43b909300b21f59327" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 36963 } }
I20240502 14:15:05.951092 18868 raft_consensus.cc:374] T a8db8a87a56b4227941f10ccd88fc0fe P d9f992bf75594e43b909300b21f59327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.951185 18868 raft_consensus.cc:729] T a8db8a87a56b4227941f10ccd88fc0fe P d9f992bf75594e43b909300b21f59327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f992bf75594e43b909300b21f59327, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.951153 18866 consensus_queue.cc:260] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1708653910d94234bd9fd7865428025c" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 36593 } } peers { permanent_uuid: "890d461ec94647c88541b1165deee654" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39651 } } peers { permanent_uuid: "d9f992bf75594e43b909300b21f59327" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 36963 } }
I20240502 14:15:05.951387 18868 consensus_queue.cc:260] T a8db8a87a56b4227941f10ccd88fc0fe P d9f992bf75594e43b909300b21f59327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1708653910d94234bd9fd7865428025c" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 36593 } } peers { permanent_uuid: "890d461ec94647c88541b1165deee654" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39651 } } peers { permanent_uuid: "d9f992bf75594e43b909300b21f59327" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 36963 } }
I20240502 14:15:05.951969 18866 ts_tablet_manager.cc:1398] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:05.952602 18868 ts_tablet_manager.cc:1398] T a8db8a87a56b4227941f10ccd88fc0fe P d9f992bf75594e43b909300b21f59327: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:06.108855 18872 raft_consensus.cc:482] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:06.109058 18872 raft_consensus.cc:504] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1708653910d94234bd9fd7865428025c" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 36593 } } peers { permanent_uuid: "890d461ec94647c88541b1165deee654" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39651 } } peers { permanent_uuid: "d9f992bf75594e43b909300b21f59327" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 36963 } }
I20240502 14:15:06.110054 18872 leader_election.cc:288] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1708653910d94234bd9fd7865428025c (127.15.74.65:36593), d9f992bf75594e43b909300b21f59327 (127.15.74.66:36963)
I20240502 14:15:06.117491 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8db8a87a56b4227941f10ccd88fc0fe" candidate_uuid: "890d461ec94647c88541b1165deee654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1708653910d94234bd9fd7865428025c" is_pre_election: true
I20240502 14:15:06.117535 18745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8db8a87a56b4227941f10ccd88fc0fe" candidate_uuid: "890d461ec94647c88541b1165deee654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f992bf75594e43b909300b21f59327" is_pre_election: true
I20240502 14:15:06.117736 18670 raft_consensus.cc:2454] T a8db8a87a56b4227941f10ccd88fc0fe P 1708653910d94234bd9fd7865428025c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 890d461ec94647c88541b1165deee654 in term 0.
I20240502 14:15:06.117779 18745 raft_consensus.cc:2454] T a8db8a87a56b4227941f10ccd88fc0fe P d9f992bf75594e43b909300b21f59327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 890d461ec94647c88541b1165deee654 in term 0.
I20240502 14:15:06.118182 18783 leader_election.cc:302] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1708653910d94234bd9fd7865428025c, 890d461ec94647c88541b1165deee654, d9f992bf75594e43b909300b21f59327; no voters:
I20240502 14:15:06.118465 18872 raft_consensus.cc:2781] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:06.118568 18872 raft_consensus.cc:482] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:06.118657 18872 raft_consensus.cc:3037] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.120791 18872 raft_consensus.cc:504] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1708653910d94234bd9fd7865428025c" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 36593 } } peers { permanent_uuid: "890d461ec94647c88541b1165deee654" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39651 } } peers { permanent_uuid: "d9f992bf75594e43b909300b21f59327" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 36963 } }
I20240502 14:15:06.121367 18872 leader_election.cc:288] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [CANDIDATE]: Term 1 election: Requested vote from peers 1708653910d94234bd9fd7865428025c (127.15.74.65:36593), d9f992bf75594e43b909300b21f59327 (127.15.74.66:36963)
I20240502 14:15:06.121790 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8db8a87a56b4227941f10ccd88fc0fe" candidate_uuid: "890d461ec94647c88541b1165deee654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1708653910d94234bd9fd7865428025c"
I20240502 14:15:06.121848 18745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8db8a87a56b4227941f10ccd88fc0fe" candidate_uuid: "890d461ec94647c88541b1165deee654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f992bf75594e43b909300b21f59327"
I20240502 14:15:06.121974 18670 raft_consensus.cc:3037] T a8db8a87a56b4227941f10ccd88fc0fe P 1708653910d94234bd9fd7865428025c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.122033 18745 raft_consensus.cc:3037] T a8db8a87a56b4227941f10ccd88fc0fe P d9f992bf75594e43b909300b21f59327 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.123909 18670 raft_consensus.cc:2454] T a8db8a87a56b4227941f10ccd88fc0fe P 1708653910d94234bd9fd7865428025c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 890d461ec94647c88541b1165deee654 in term 1.
I20240502 14:15:06.123971 18745 raft_consensus.cc:2454] T a8db8a87a56b4227941f10ccd88fc0fe P d9f992bf75594e43b909300b21f59327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 890d461ec94647c88541b1165deee654 in term 1.
I20240502 14:15:06.124303 18782 leader_election.cc:302] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 890d461ec94647c88541b1165deee654, d9f992bf75594e43b909300b21f59327; no voters:
I20240502 14:15:06.124567 18872 raft_consensus.cc:2781] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:06.124944 18872 raft_consensus.cc:686] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [term 1 LEADER]: Becoming Leader. State: Replica: 890d461ec94647c88541b1165deee654, State: Running, Role: LEADER
I20240502 14:15:06.125200 18872 consensus_queue.cc:237] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1708653910d94234bd9fd7865428025c" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 36593 } } peers { permanent_uuid: "890d461ec94647c88541b1165deee654" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39651 } } peers { permanent_uuid: "d9f992bf75594e43b909300b21f59327" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 36963 } }
I20240502 14:15:06.127449 18568 catalog_manager.cc:5484] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 reported cstate change: term changed from 0 to 1, leader changed from <none> to 890d461ec94647c88541b1165deee654 (127.15.74.67). New cstate: current_term: 1 leader_uuid: "890d461ec94647c88541b1165deee654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1708653910d94234bd9fd7865428025c" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "890d461ec94647c88541b1165deee654" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9f992bf75594e43b909300b21f59327" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 36963 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:06.541427 18872 consensus_queue.cc:1035] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1708653910d94234bd9fd7865428025c" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:06.546819 18881 consensus_queue.cc:1035] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9f992bf75594e43b909300b21f59327" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 36963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:06.686278 18616 placement_policy_util.cc:407] tablet a8db8a87a56b4227941f10ccd88fc0fe: detected majority of replicas (2 of 3) at location L0
I20240502 14:15:06.686580 18621 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:06.687011 18621 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:15:06.687136 18621 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:07.689260 18616 placement_policy_util.cc:407] tablet a8db8a87a56b4227941f10ccd88fc0fe: detected majority of replicas (2 of 3) at location L0
I20240502 14:15:08.687706 18621 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:08.688207 18621 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:15:08.688328 18621 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:08.691610 18616 placement_policy_util.cc:407] tablet a8db8a87a56b4227941f10ccd88fc0fe: detected majority of replicas (2 of 3) at location L0
I20240502 14:15:09.693948 18616 placement_policy_util.cc:407] tablet a8db8a87a56b4227941f10ccd88fc0fe: detected majority of replicas (2 of 3) at location L0
I20240502 14:15:10.688783 18621 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:10.689265 18621 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:15:10.689385 18621 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:10.696187 18616 placement_policy_util.cc:407] tablet a8db8a87a56b4227941f10ccd88fc0fe: detected majority of replicas (2 of 3) at location L0
I20240502 14:15:11.698696 18616 placement_policy_util.cc:407] tablet a8db8a87a56b4227941f10ccd88fc0fe: detected majority of replicas (2 of 3) at location L0
I20240502 14:15:12.689831 18621 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:12.690276 18621 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:15:12.690375 18621 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:12.700920 18616 placement_policy_util.cc:407] tablet a8db8a87a56b4227941f10ccd88fc0fe: detected majority of replicas (2 of 3) at location L0
I20240502 14:15:13.703342 18616 placement_policy_util.cc:407] tablet a8db8a87a56b4227941f10ccd88fc0fe: detected majority of replicas (2 of 3) at location L0
I20240502 14:15:14.690793 18621 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:14.691270 18621 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:15:14.691385 18621 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:14.705518 18616 placement_policy_util.cc:407] tablet a8db8a87a56b4227941f10ccd88fc0fe: detected majority of replicas (2 of 3) at location L0
I20240502 14:15:15.707772 18616 placement_policy_util.cc:407] tablet a8db8a87a56b4227941f10ccd88fc0fe: detected majority of replicas (2 of 3) at location L0
I20240502 14:15:16.691795 18621 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:16.692252 18621 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:15:16.692365 18621 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:16.710361 18616 placement_policy_util.cc:407] tablet a8db8a87a56b4227941f10ccd88fc0fe: detected majority of replicas (2 of 3) at location L0
I20240502 14:15:17.220690 15657 tablet_server.cc:178] TabletServer@127.15.74.65:0 shutting down...
W20240502 14:15:17.231814 18783 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
I20240502 14:15:17.232468 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:15:17.232745 15657 tablet_replica.cc:330] T a8db8a87a56b4227941f10ccd88fc0fe P 1708653910d94234bd9fd7865428025c: stopping tablet replica
I20240502 14:15:17.233001 15657 raft_consensus.cc:2229] T a8db8a87a56b4227941f10ccd88fc0fe P 1708653910d94234bd9fd7865428025c [term 1 FOLLOWER]: Raft consensus shutting down.
W20240502 14:15:17.233057 18783 consensus_peers.cc:485] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 -> Peer 1708653910d94234bd9fd7865428025c (127.15.74.65:36593): Couldn't send request to peer 1708653910d94234bd9fd7865428025c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:17.233194 15657 raft_consensus.cc:2258] T a8db8a87a56b4227941f10ccd88fc0fe P 1708653910d94234bd9fd7865428025c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:17.247368 15657 tablet_server.cc:194] TabletServer@127.15.74.65:0 shutdown complete.
I20240502 14:15:17.250164 15657 tablet_server.cc:178] TabletServer@127.15.74.66:0 shutting down...
I20240502 14:15:17.257709 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:15:17.257946 15657 tablet_replica.cc:330] T a8db8a87a56b4227941f10ccd88fc0fe P d9f992bf75594e43b909300b21f59327: stopping tablet replica
I20240502 14:15:17.258103 15657 raft_consensus.cc:2229] T a8db8a87a56b4227941f10ccd88fc0fe P d9f992bf75594e43b909300b21f59327 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:17.258270 15657 raft_consensus.cc:2258] T a8db8a87a56b4227941f10ccd88fc0fe P d9f992bf75594e43b909300b21f59327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:17.271612 15657 tablet_server.cc:194] TabletServer@127.15.74.66:0 shutdown complete.
I20240502 14:15:17.274370 15657 tablet_server.cc:178] TabletServer@127.15.74.67:0 shutting down...
I20240502 14:15:17.281461 15657 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:15:17.281699 15657 tablet_replica.cc:330] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654: stopping tablet replica
I20240502 14:15:17.281884 15657 raft_consensus.cc:2229] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:17.282174 15657 raft_consensus.cc:2258] T a8db8a87a56b4227941f10ccd88fc0fe P 890d461ec94647c88541b1165deee654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:17.295483 15657 tablet_server.cc:194] TabletServer@127.15.74.67:0 shutdown complete.
I20240502 14:15:17.298494 15657 master.cc:537] Master@127.15.74.126:46613 shutting down...
I20240502 14:15:17.306217 15657 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:17.306403 15657 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:17.306501 15657 tablet_replica.cc:330] T 00000000000000000000000000000000 P b1c2261244d043229835a06bb3127c57: stopping tablet replica
I20240502 14:15:17.319607 15657 master.cc:552] Master@127.15.74.126:46613 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12691 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240502 14:15:17.330293 15657 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.15.74.126:42275
I20240502 14:15:17.330790 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:17.333274 18889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:17.334007 18890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:17.334318 18892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:17.334959 15657 server_base.cc:1034] running on GCE node
I20240502 14:15:17.335201 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:15:17.335264 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:15:17.335323 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659317335310 us; error 0 us; skew 500 ppm
I20240502 14:15:17.335522 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:17.342723 15657 webserver.cc:460] Webserver started at http://127.15.74.126:45939/ using document root <none> and password file <none>
I20240502 14:15:17.342943 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:17.343010 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:17.343133 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:17.343839 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/master-0-root/instance:
uuid: "9f99fb547d9a4761bf0d901a52218086"
format_stamp: "Formatted at 2024-05-02 14:15:17 on dist-test-slave-wbm6"
I20240502 14:15:17.346400 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:17.347906 18897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:17.348265 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:15:17.348400 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/master-0-root
uuid: "9f99fb547d9a4761bf0d901a52218086"
format_stamp: "Formatted at 2024-05-02 14:15:17 on dist-test-slave-wbm6"
I20240502 14:15:17.348523 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:17.369827 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:17.370373 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:17.380851 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.126:42275
I20240502 14:15:17.380880 18948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.126:42275 every 8 connection(s)
I20240502 14:15:17.382258 18949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.386101 18949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086: Bootstrap starting.
I20240502 14:15:17.387182 18949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.388631 18949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086: No bootstrap required, opened a new log
I20240502 14:15:17.389143 18949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f99fb547d9a4761bf0d901a52218086" member_type: VOTER }
I20240502 14:15:17.389268 18949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.389322 18949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f99fb547d9a4761bf0d901a52218086, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.389520 18949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f99fb547d9a4761bf0d901a52218086" member_type: VOTER }
I20240502 14:15:17.389642 18949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:15:17.389703 18949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:15:17.389797 18949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.391511 18949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f99fb547d9a4761bf0d901a52218086" member_type: VOTER }
I20240502 14:15:17.391697 18949 leader_election.cc:302] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f99fb547d9a4761bf0d901a52218086; no voters:
I20240502 14:15:17.392095 18949 leader_election.cc:288] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:15:17.392271 18952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:17.392796 18952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [term 1 LEADER]: Becoming Leader. State: Replica: 9f99fb547d9a4761bf0d901a52218086, State: Running, Role: LEADER
I20240502 14:15:17.392849 18949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:15:17.393049 18952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f99fb547d9a4761bf0d901a52218086" member_type: VOTER }
I20240502 14:15:17.393872 18953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f99fb547d9a4761bf0d901a52218086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f99fb547d9a4761bf0d901a52218086" member_type: VOTER } }
I20240502 14:15:17.393916 18954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f99fb547d9a4761bf0d901a52218086. Latest consensus state: current_term: 1 leader_uuid: "9f99fb547d9a4761bf0d901a52218086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f99fb547d9a4761bf0d901a52218086" member_type: VOTER } }
I20240502 14:15:17.394187 18954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [sys.catalog]: This master's current role is: LEADER
I20240502 14:15:17.394728 18953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086 [sys.catalog]: This master's current role is: LEADER
I20240502 14:15:17.395495 18958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:15:17.397305 18958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:15:17.397614 15657 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:15:17.399799 18958 catalog_manager.cc:1347] Generated new cluster ID: da09348f8eac48c7b643a45655291daa
I20240502 14:15:17.399871 18958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:15:17.424580 18958 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:15:17.425403 18958 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:15:17.438654 18958 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9f99fb547d9a4761bf0d901a52218086: Generated new TSK 0
I20240502 14:15:17.438867 18958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:15:17.462433 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:17.465143 18970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:17.466055 18973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:17.466167 15657 server_base.cc:1034] running on GCE node
W20240502 14:15:17.466171 18971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:17.466599 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:15:17.466675 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:15:17.466745 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659317466726 us; error 0 us; skew 500 ppm
I20240502 14:15:17.466895 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:17.468103 15657 webserver.cc:460] Webserver started at http://127.15.74.65:36417/ using document root <none> and password file <none>
I20240502 14:15:17.468358 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:17.468443 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:17.468596 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:17.469233 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-0-root/instance:
uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
format_stamp: "Formatted at 2024-05-02 14:15:17 on dist-test-slave-wbm6"
I20240502 14:15:17.471801 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 14:15:17.473340 18978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:17.473706 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 14:15:17.473829 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-0-root
uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
format_stamp: "Formatted at 2024-05-02 14:15:17 on dist-test-slave-wbm6"
I20240502 14:15:17.473961 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:17.501233 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:17.502049 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:17.502693 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:17.503594 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:17.503667 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:17.503772 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:17.503830 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:17.515446 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.65:41141
I20240502 14:15:17.515499 19040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.65:41141 every 8 connection(s)
I20240502 14:15:17.517055 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:17.519918 19045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:17.520514 19046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:17.520742 19048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:17.521075 15657 server_base.cc:1034] running on GCE node
I20240502 14:15:17.521370 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:15:17.521466 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:15:17.521539 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659317521513 us; error 0 us; skew 500 ppm
I20240502 14:15:17.521708 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:17.523795 15657 webserver.cc:460] Webserver started at http://127.15.74.66:42009/ using document root <none> and password file <none>
I20240502 14:15:17.524075 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:17.524123 19041 heartbeater.cc:347] Connected to a master server at 127.15.74.126:42275
I20240502 14:15:17.524190 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:17.524269 19041 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:17.524363 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:17.524732 19041 heartbeater.cc:511] Master 127.15.74.126:42275 requested a full tablet report, sending...
I20240502 14:15:17.525154 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-1-root/instance:
uuid: "80654f573c764bc18e65d583b751997f"
format_stamp: "Formatted at 2024-05-02 14:15:17 on dist-test-slave-wbm6"
I20240502 14:15:17.525640 18914 ts_manager.cc:194] Registered new tserver with Master: 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141)
I20240502 14:15:17.526702 18914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51740
I20240502 14:15:17.527976 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20240502 14:15:17.529333 19053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:17.529731 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 14:15:17.529848 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-1-root
uuid: "80654f573c764bc18e65d583b751997f"
format_stamp: "Formatted at 2024-05-02 14:15:17 on dist-test-slave-wbm6"
I20240502 14:15:17.529964 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:17.558254 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:17.558764 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:17.559257 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:17.560055 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:17.560112 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:17.560192 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:17.560228 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:17.571063 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.66:44583
I20240502 14:15:17.571082 19115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.66:44583 every 8 connection(s)
I20240502 14:15:17.572563 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:17.575982 19120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:17.576138 19119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:17.576745 19122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:17.577220 15657 server_base.cc:1034] running on GCE node
I20240502 14:15:17.577675 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:15:17.577744 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:15:17.577769 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659317577766 us; error 0 us; skew 500 ppm
I20240502 14:15:17.577909 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:17.578925 19116 heartbeater.cc:347] Connected to a master server at 127.15.74.126:42275
I20240502 14:15:17.579046 19116 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:17.579322 15657 webserver.cc:460] Webserver started at http://127.15.74.67:33261/ using document root <none> and password file <none>
I20240502 14:15:17.579479 19116 heartbeater.cc:511] Master 127.15.74.126:42275 requested a full tablet report, sending...
I20240502 14:15:17.579571 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:17.579658 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:17.579772 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:17.580369 18914 ts_manager.cc:194] Registered new tserver with Master: 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:17.580435 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-2-root/instance:
uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
format_stamp: "Formatted at 2024-05-02 14:15:17 on dist-test-slave-wbm6"
I20240502 14:15:17.581133 18914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51744
I20240502 14:15:17.582823 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:17.584205 19127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:17.584555 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:15:17.584635 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-2-root
uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
format_stamp: "Formatted at 2024-05-02 14:15:17 on dist-test-slave-wbm6"
I20240502 14:15:17.584717 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:17.597136 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:17.597590 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:17.598024 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:17.598718 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:17.598768 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:17.598820 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:17.598850 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:17.609001 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.67:39767
I20240502 14:15:17.609025 19189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.67:39767 every 8 connection(s)
I20240502 14:15:17.615564 19190 heartbeater.cc:347] Connected to a master server at 127.15.74.126:42275
I20240502 14:15:17.615691 19190 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:17.615988 19190 heartbeater.cc:511] Master 127.15.74.126:42275 requested a full tablet report, sending...
I20240502 14:15:17.616776 18914 ts_manager.cc:194] Registered new tserver with Master: 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:17.617491 18914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51746
I20240502 14:15:17.617666 15657 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007827833s
I20240502 14:15:18.528211 19041 heartbeater.cc:503] Master 127.15.74.126:42275 was elected leader, sending a full tablet report...
I20240502 14:15:18.582283 19116 heartbeater.cc:503] Master 127.15.74.126:42275 was elected leader, sending a full tablet report...
I20240502 14:15:18.618630 19190 heartbeater.cc:503] Master 127.15.74.126:42275 was elected leader, sending a full tablet report...
I20240502 14:15:18.645123 15657 test_util.cc:268] Using random seed: 999899489
I20240502 14:15:18.656745 18914 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51756:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:15:18.657502 18914 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:15:18.691390 19081 tablet_service.cc:1467] Processing CreateTablet for tablet 154f396806954d2eb24baa4a86ba876d (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 14:15:18.691597 19150 tablet_service.cc:1467] Processing CreateTablet for tablet 342705e9da8043ae9aebe50310650d2c (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 14:15:18.692075 19150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342705e9da8043ae9aebe50310650d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.692494 19080 tablet_service.cc:1467] Processing CreateTablet for tablet 76a8747fc61249759eb84a0110df4efb (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 14:15:18.692090 19081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 154f396806954d2eb24baa4a86ba876d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.692996 19080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a8747fc61249759eb84a0110df4efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.693274 19146 tablet_service.cc:1467] Processing CreateTablet for tablet 618d11b0b08e4d49ac15d443acd10e2c (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 14:15:18.693635 19146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618d11b0b08e4d49ac15d443acd10e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.695772 19147 tablet_service.cc:1467] Processing CreateTablet for tablet 99128e8f96dc4e1582ec51cbcb8085e0 (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 14:15:18.696153 19147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99128e8f96dc4e1582ec51cbcb8085e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.699641 19079 tablet_service.cc:1467] Processing CreateTablet for tablet f260d59158a344beb686ea0f9746f743 (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 14:15:18.699895 19148 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef2dcb2456949cfa34b5c2e8b0d2e6d (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 14:15:18.700179 19079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f260d59158a344beb686ea0f9746f743. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.700383 19148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef2dcb2456949cfa34b5c2e8b0d2e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.695787 19078 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb757082b084620af257fb553d2dd4b (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 14:15:18.704034 19078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb757082b084620af257fb553d2dd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.703878 19149 tablet_service.cc:1467] Processing CreateTablet for tablet 8f465cbf06f4420c8642e8092ce8e1ed (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 14:15:18.704383 19149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f465cbf06f4420c8642e8092ce8e1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.707847 19151 tablet_service.cc:1467] Processing CreateTablet for tablet 567ee47202bd46048e318b4dbbf3ff03 (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 14:15:18.708299 19151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567ee47202bd46048e318b4dbbf3ff03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.711879 19152 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb757082b084620af257fb553d2dd4b (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 14:15:18.712425 19152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb757082b084620af257fb553d2dd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.715910 19153 tablet_service.cc:1467] Processing CreateTablet for tablet f260d59158a344beb686ea0f9746f743 (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 14:15:18.716450 19153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f260d59158a344beb686ea0f9746f743. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.719866 19154 tablet_service.cc:1467] Processing CreateTablet for tablet 76a8747fc61249759eb84a0110df4efb (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 14:15:18.720387 19154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a8747fc61249759eb84a0110df4efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.726445 19077 tablet_service.cc:1467] Processing CreateTablet for tablet 567ee47202bd46048e318b4dbbf3ff03 (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 14:15:18.726979 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567ee47202bd46048e318b4dbbf3ff03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.730360 19155 tablet_service.cc:1467] Processing CreateTablet for tablet 154f396806954d2eb24baa4a86ba876d (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 14:15:18.730882 19155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 154f396806954d2eb24baa4a86ba876d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.733464 19079 tablet_service.cc:1467] Processing CreateTablet for tablet 8f465cbf06f4420c8642e8092ce8e1ed (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 14:15:18.733875 19079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f465cbf06f4420c8642e8092ce8e1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.739784 19080 tablet_service.cc:1467] Processing CreateTablet for tablet 342705e9da8043ae9aebe50310650d2c (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 14:15:18.740249 19080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342705e9da8043ae9aebe50310650d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.747453 19076 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef2dcb2456949cfa34b5c2e8b0d2e6d (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 14:15:18.747988 19076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef2dcb2456949cfa34b5c2e8b0d2e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.751070 19075 tablet_service.cc:1467] Processing CreateTablet for tablet 73c203a4764245dba0944537c7d10129 (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 14:15:18.751252 19074 tablet_service.cc:1467] Processing CreateTablet for tablet 35d01415abc44ec4b52577992892c5b1 (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 14:15:18.751505 19075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c203a4764245dba0944537c7d10129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.752784 19079 tablet_service.cc:1467] Processing CreateTablet for tablet 618d11b0b08e4d49ac15d443acd10e2c (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 14:15:18.753222 19079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618d11b0b08e4d49ac15d443acd10e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.754185 19080 tablet_service.cc:1467] Processing CreateTablet for tablet 99128e8f96dc4e1582ec51cbcb8085e0 (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 14:15:18.754585 19080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99128e8f96dc4e1582ec51cbcb8085e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.760427 19147 tablet_service.cc:1467] Processing CreateTablet for tablet 73c203a4764245dba0944537c7d10129 (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 14:15:18.760870 19147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c203a4764245dba0944537c7d10129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.767021 19210 tablet_bootstrap.cc:492] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78: Bootstrap starting.
I20240502 14:15:18.766968 19150 tablet_service.cc:1467] Processing CreateTablet for tablet 35d01415abc44ec4b52577992892c5b1 (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 14:15:18.767280 19150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35d01415abc44ec4b52577992892c5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.768673 19210 tablet_bootstrap.cc:654] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.769668 19074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35d01415abc44ec4b52577992892c5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.788740 19211 tablet_bootstrap.cc:492] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f: Bootstrap starting.
I20240502 14:15:18.791178 19211 tablet_bootstrap.cc:654] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.792565 19210 tablet_bootstrap.cc:492] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78: No bootstrap required, opened a new log
I20240502 14:15:18.792758 19210 ts_tablet_manager.cc:1367] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.005s
I20240502 14:15:18.793182 19211 tablet_bootstrap.cc:492] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f: No bootstrap required, opened a new log
I20240502 14:15:18.793319 19211 ts_tablet_manager.cc:1367] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:15:18.793460 19210 raft_consensus.cc:348] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.793633 19210 raft_consensus.cc:374] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.793700 19210 raft_consensus.cc:729] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.793923 19211 raft_consensus.cc:348] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.793958 19210 consensus_queue.cc:260] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.794173 19211 raft_consensus.cc:374] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.794296 19211 raft_consensus.cc:729] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.794517 19211 consensus_queue.cc:260] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.795255 19211 ts_tablet_manager.cc:1398] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:18.798197 19211 tablet_bootstrap.cc:492] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f: Bootstrap starting.
I20240502 14:15:18.799206 19005 tablet_service.cc:1467] Processing CreateTablet for tablet 76a8747fc61249759eb84a0110df4efb (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 14:15:18.799715 19211 tablet_bootstrap.cc:654] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.799779 19005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a8747fc61249759eb84a0110df4efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.800040 19006 tablet_service.cc:1467] Processing CreateTablet for tablet 154f396806954d2eb24baa4a86ba876d (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 14:15:18.800472 19006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 154f396806954d2eb24baa4a86ba876d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.803884 19004 tablet_service.cc:1467] Processing CreateTablet for tablet f260d59158a344beb686ea0f9746f743 (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 14:15:18.804041 19210 ts_tablet_manager.cc:1398] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
I20240502 14:15:18.804313 19004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f260d59158a344beb686ea0f9746f743. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.804327 19210 tablet_bootstrap.cc:492] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78: Bootstrap starting.
I20240502 14:15:18.805846 19210 tablet_bootstrap.cc:654] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.807894 19211 tablet_bootstrap.cc:492] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f: No bootstrap required, opened a new log
I20240502 14:15:18.807792 19003 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb757082b084620af257fb553d2dd4b (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 14:15:18.808279 19003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb757082b084620af257fb553d2dd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.808641 19001 tablet_service.cc:1467] Processing CreateTablet for tablet 342705e9da8043ae9aebe50310650d2c (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 14:15:18.809041 19001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342705e9da8043ae9aebe50310650d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.809808 19218 tablet_bootstrap.cc:492] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Bootstrap starting.
I20240502 14:15:18.811884 19218 tablet_bootstrap.cc:654] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.811784 19000 tablet_service.cc:1467] Processing CreateTablet for tablet 8f465cbf06f4420c8642e8092ce8e1ed (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 14:15:18.808295 19002 tablet_service.cc:1467] Processing CreateTablet for tablet 567ee47202bd46048e318b4dbbf3ff03 (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 14:15:18.812287 19000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f465cbf06f4420c8642e8092ce8e1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.812534 19002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567ee47202bd46048e318b4dbbf3ff03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.812986 18999 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef2dcb2456949cfa34b5c2e8b0d2e6d (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 14:15:18.813397 18999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef2dcb2456949cfa34b5c2e8b0d2e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.813664 19005 tablet_service.cc:1467] Processing CreateTablet for tablet 73c203a4764245dba0944537c7d10129 (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 14:15:18.814079 19005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c203a4764245dba0944537c7d10129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.815825 18998 tablet_service.cc:1467] Processing CreateTablet for tablet 99128e8f96dc4e1582ec51cbcb8085e0 (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 14:15:18.816190 18998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99128e8f96dc4e1582ec51cbcb8085e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.820564 19211 ts_tablet_manager.cc:1367] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.005s
I20240502 14:15:18.823796 19002 tablet_service.cc:1467] Processing CreateTablet for tablet 35d01415abc44ec4b52577992892c5b1 (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 14:15:18.824143 19002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35d01415abc44ec4b52577992892c5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.828908 18997 tablet_service.cc:1467] Processing CreateTablet for tablet 618d11b0b08e4d49ac15d443acd10e2c (DEFAULT_TABLE table=test-workload [id=682791e08deb409e86af13a3a7eb1967]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 14:15:18.829741 18997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618d11b0b08e4d49ac15d443acd10e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.834949 19218 tablet_bootstrap.cc:492] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: No bootstrap required, opened a new log
I20240502 14:15:18.835134 19218 ts_tablet_manager.cc:1367] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240502 14:15:18.836126 19218 raft_consensus.cc:348] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.836510 19218 raft_consensus.cc:374] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.836633 19218 raft_consensus.cc:729] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.836896 19218 consensus_queue.cc:260] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.840250 19211 raft_consensus.cc:348] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.840920 19211 raft_consensus.cc:374] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.840993 19211 raft_consensus.cc:729] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.841187 19211 consensus_queue.cc:260] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.841563 19211 ts_tablet_manager.cc:1398] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20240502 14:15:18.841810 19211 tablet_bootstrap.cc:492] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f: Bootstrap starting.
I20240502 14:15:18.845198 19211 tablet_bootstrap.cc:654] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.846202 19218 ts_tablet_manager.cc:1398] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240502 14:15:18.846500 19218 tablet_bootstrap.cc:492] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Bootstrap starting.
I20240502 14:15:18.847968 19218 tablet_bootstrap.cc:654] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.849282 19210 tablet_bootstrap.cc:492] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78: No bootstrap required, opened a new log
I20240502 14:15:18.849429 19210 ts_tablet_manager.cc:1367] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent bootstrapping tablet: real 0.045s user 0.016s sys 0.024s
I20240502 14:15:18.850010 19218 tablet_bootstrap.cc:492] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: No bootstrap required, opened a new log
I20240502 14:15:18.850127 19218 ts_tablet_manager.cc:1367] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:15:18.850231 19210 raft_consensus.cc:348] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.850502 19210 raft_consensus.cc:374] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.850649 19210 raft_consensus.cc:729] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.850827 19218 raft_consensus.cc:348] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.851028 19218 raft_consensus.cc:374] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.851112 19218 raft_consensus.cc:729] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.851243 19210 consensus_queue.cc:260] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.851477 19218 consensus_queue.cc:260] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.851814 19210 ts_tablet_manager.cc:1398] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:18.852121 19210 tablet_bootstrap.cc:492] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78: Bootstrap starting.
I20240502 14:15:18.853559 19210 tablet_bootstrap.cc:654] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.853612 19211 tablet_bootstrap.cc:492] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f: No bootstrap required, opened a new log
I20240502 14:15:18.853768 19211 ts_tablet_manager.cc:1367] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20240502 14:15:18.854303 19211 raft_consensus.cc:348] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.854532 19211 raft_consensus.cc:374] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.854635 19211 raft_consensus.cc:729] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.854899 19211 consensus_queue.cc:260] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.855281 19210 tablet_bootstrap.cc:492] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78: No bootstrap required, opened a new log
I20240502 14:15:18.855412 19218 ts_tablet_manager.cc:1398] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240502 14:15:18.855487 19210 ts_tablet_manager.cc:1367] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 14:15:18.856295 19211 ts_tablet_manager.cc:1398] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:18.856225 19210 raft_consensus.cc:348] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.856519 19210 raft_consensus.cc:374] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.856633 19211 tablet_bootstrap.cc:492] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f: Bootstrap starting.
I20240502 14:15:18.856650 19210 raft_consensus.cc:729] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.857039 19210 consensus_queue.cc:260] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.857542 19210 ts_tablet_manager.cc:1398] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:18.857872 19210 tablet_bootstrap.cc:492] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78: Bootstrap starting.
I20240502 14:15:18.858942 19211 tablet_bootstrap.cc:654] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.859347 19210 tablet_bootstrap.cc:654] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.860680 19211 tablet_bootstrap.cc:492] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f: No bootstrap required, opened a new log
I20240502 14:15:18.860831 19211 ts_tablet_manager.cc:1367] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:18.861320 19218 tablet_bootstrap.cc:492] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Bootstrap starting.
I20240502 14:15:18.861644 19210 tablet_bootstrap.cc:492] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78: No bootstrap required, opened a new log
I20240502 14:15:18.861634 19211 raft_consensus.cc:348] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.861819 19211 raft_consensus.cc:374] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.861867 19210 ts_tablet_manager.cc:1367] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:15:18.861905 19211 raft_consensus.cc:729] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.862190 19211 consensus_queue.cc:260] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.862700 19210 raft_consensus.cc:348] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.862979 19210 raft_consensus.cc:374] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.863026 19211 ts_tablet_manager.cc:1398] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:18.863086 19210 raft_consensus.cc:729] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.863293 19211 tablet_bootstrap.cc:492] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f: Bootstrap starting.
I20240502 14:15:18.863324 19210 consensus_queue.cc:260] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.863530 19218 tablet_bootstrap.cc:654] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.864069 19210 ts_tablet_manager.cc:1398] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:18.864413 19210 tablet_bootstrap.cc:492] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78: Bootstrap starting.
I20240502 14:15:18.866067 19210 tablet_bootstrap.cc:654] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.866626 19218 tablet_bootstrap.cc:492] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: No bootstrap required, opened a new log
I20240502 14:15:18.866775 19218 ts_tablet_manager.cc:1367] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240502 14:15:18.867579 19218 raft_consensus.cc:348] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.867825 19218 raft_consensus.cc:374] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.867919 19218 raft_consensus.cc:729] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.868144 19218 consensus_queue.cc:260] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.868327 19211 tablet_bootstrap.cc:654] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.868767 19218 ts_tablet_manager.cc:1398] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:18.868990 19210 tablet_bootstrap.cc:492] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78: No bootstrap required, opened a new log
I20240502 14:15:18.869083 19218 tablet_bootstrap.cc:492] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Bootstrap starting.
I20240502 14:15:18.869172 19210 ts_tablet_manager.cc:1367] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:15:18.870008 19210 raft_consensus.cc:348] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.870302 19210 raft_consensus.cc:374] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.870461 19210 raft_consensus.cc:729] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.870790 19210 consensus_queue.cc:260] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.871208 19218 tablet_bootstrap.cc:654] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.872186 19210 ts_tablet_manager.cc:1398] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:15:18.872541 19210 tablet_bootstrap.cc:492] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78: Bootstrap starting.
I20240502 14:15:18.874159 19210 tablet_bootstrap.cc:654] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.877671 19211 tablet_bootstrap.cc:492] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f: No bootstrap required, opened a new log
I20240502 14:15:18.877919 19211 ts_tablet_manager.cc:1367] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20240502 14:15:18.878832 19211 raft_consensus.cc:348] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.879191 19211 raft_consensus.cc:374] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.879339 19211 raft_consensus.cc:729] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.879760 19211 consensus_queue.cc:260] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.880353 19210 tablet_bootstrap.cc:492] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78: No bootstrap required, opened a new log
I20240502 14:15:18.880538 19210 ts_tablet_manager.cc:1367] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20240502 14:15:18.880592 19211 ts_tablet_manager.cc:1398] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:15:18.881032 19211 tablet_bootstrap.cc:492] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f: Bootstrap starting.
I20240502 14:15:18.881292 19210 raft_consensus.cc:348] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.881639 19210 raft_consensus.cc:374] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.881794 19210 raft_consensus.cc:729] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.882051 19220 raft_consensus.cc:482] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:18.882200 19220 raft_consensus.cc:504] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.883309 19220 leader_election.cc:288] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:18.883387 19211 tablet_bootstrap.cc:654] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.885327 19218 tablet_bootstrap.cc:492] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: No bootstrap required, opened a new log
I20240502 14:15:18.885380 19210 consensus_queue.cc:260] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.885574 19218 ts_tablet_manager.cc:1367] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20240502 14:15:18.886181 19210 ts_tablet_manager.cc:1398] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:15:18.886565 19218 raft_consensus.cc:348] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.886840 19218 raft_consensus.cc:374] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.886955 19218 raft_consensus.cc:729] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.887265 19218 consensus_queue.cc:260] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.887949 19218 ts_tablet_manager.cc:1398] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:18.888306 19218 tablet_bootstrap.cc:492] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Bootstrap starting.
I20240502 14:15:18.890494 19218 tablet_bootstrap.cc:654] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.886575 19210 tablet_bootstrap.cc:492] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78: Bootstrap starting.
I20240502 14:15:18.893036 19211 tablet_bootstrap.cc:492] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f: No bootstrap required, opened a new log
I20240502 14:15:18.893261 19211 ts_tablet_manager.cc:1367] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.001s
I20240502 14:15:18.893709 19210 tablet_bootstrap.cc:654] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.893958 19165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" is_pre_election: true
I20240502 14:15:18.894193 19165 raft_consensus.cc:2454] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 0.
I20240502 14:15:18.894140 19211 raft_consensus.cc:348] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.894347 19211 raft_consensus.cc:374] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.894433 19211 raft_consensus.cc:729] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.894618 18979 leader_election.cc:302] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78; no voters:
I20240502 14:15:18.894717 19211 consensus_queue.cc:260] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.894937 19220 raft_consensus.cc:2781] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:18.895067 19220 raft_consensus.cc:482] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:18.895140 19220 raft_consensus.cc:3037] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.895670 19211 ts_tablet_manager.cc:1398] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:18.896034 19211 tablet_bootstrap.cc:492] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f: Bootstrap starting.
I20240502 14:15:18.898082 19220 raft_consensus.cc:504] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.899108 19220 leader_election.cc:288] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 election: Requested vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:18.899608 19165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:18.899798 19165 raft_consensus.cc:3037] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.900028 19218 tablet_bootstrap.cc:492] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: No bootstrap required, opened a new log
I20240502 14:15:18.900182 19218 ts_tablet_manager.cc:1367] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 14:15:18.900964 19218 raft_consensus.cc:348] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.901185 19218 raft_consensus.cc:374] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.901268 19218 raft_consensus.cc:729] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.901518 19218 consensus_queue.cc:260] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.903015 19218 ts_tablet_manager.cc:1398] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:18.903049 19165 raft_consensus.cc:2454] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 1.
I20240502 14:15:18.903517 19214 raft_consensus.cc:482] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:18.903576 18979 leader_election.cc:302] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78; no voters:
I20240502 14:15:18.903676 19214 raft_consensus.cc:504] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.904693 19214 leader_election.cc:288] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:18.908455 19220 raft_consensus.cc:2781] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:18.909034 19220 raft_consensus.cc:686] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Becoming Leader. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:18.909502 19220 consensus_queue.cc:237] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.903332 19218 tablet_bootstrap.cc:492] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Bootstrap starting.
I20240502 14:15:18.918236 19218 tablet_bootstrap.cc:654] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.898092 19211 tablet_bootstrap.cc:654] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.921270 19210 tablet_bootstrap.cc:492] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78: No bootstrap required, opened a new log
I20240502 14:15:18.921501 19210 ts_tablet_manager.cc:1367] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent bootstrapping tablet: real 0.035s user 0.018s sys 0.009s
I20240502 14:15:18.921420 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f" is_pre_election: true
I20240502 14:15:18.922292 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:18.922457 19210 raft_consensus.cc:348] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.922686 19210 raft_consensus.cc:374] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.922778 19210 raft_consensus.cc:729] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Initialized, Role: FOLLOWER
W20240502 14:15:18.922829 18982 leader_election.cc:341] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80654f573c764bc18e65d583b751997f (127.15.74.66:44583): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:18.923045 19210 consensus_queue.cc:260] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
W20240502 14:15:18.923475 18982 leader_election.cc:341] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 80654f573c764bc18e65d583b751997f (127.15.74.66:44583): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:18.924348 19210 ts_tablet_manager.cc:1398] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:15:18.924659 19210 tablet_bootstrap.cc:492] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78: Bootstrap starting.
I20240502 14:15:18.926350 19210 tablet_bootstrap.cc:654] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.933338 19218 tablet_bootstrap.cc:492] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: No bootstrap required, opened a new log
I20240502 14:15:18.933609 19218 ts_tablet_manager.cc:1367] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.000s
I20240502 14:15:18.934231 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2dcb2456949cfa34b5c2e8b0d2e6d" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f" is_pre_election: true
W20240502 14:15:18.934788 19131 leader_election.cc:341] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80654f573c764bc18e65d583b751997f (127.15.74.66:44583): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:18.934695 19218 raft_consensus.cc:348] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.935089 19218 raft_consensus.cc:374] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.935253 19218 raft_consensus.cc:729] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.935681 19218 consensus_queue.cc:260] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.936525 19218 ts_tablet_manager.cc:1398] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:18.936950 19218 tablet_bootstrap.cc:492] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Bootstrap starting.
I20240502 14:15:18.938666 19016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2dcb2456949cfa34b5c2e8b0d2e6d" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" is_pre_election: true
I20240502 14:15:18.938522 19211 tablet_bootstrap.cc:492] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f: No bootstrap required, opened a new log
I20240502 14:15:18.939090 19211 ts_tablet_manager.cc:1367] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.005s
W20240502 14:15:18.939355 19130 leader_election.cc:341] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:18.939488 19130 leader_election.cc:302] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c60ef9c4e8f439facdfdc6fd669ae78; no voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f
I20240502 14:15:18.939755 19214 raft_consensus.cc:2726] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:18.939877 19211 raft_consensus.cc:348] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.940099 19211 raft_consensus.cc:374] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.940189 19211 raft_consensus.cc:729] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.939878 18912 catalog_manager.cc:5484] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65). New cstate: current_term: 1 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:18.940443 19211 consensus_queue.cc:260] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.940593 19210 tablet_bootstrap.cc:492] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78: No bootstrap required, opened a new log
I20240502 14:15:18.940768 19210 ts_tablet_manager.cc:1367] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20240502 14:15:18.941008 19211 ts_tablet_manager.cc:1398] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:18.941321 19211 tablet_bootstrap.cc:492] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f: Bootstrap starting.
I20240502 14:15:18.941720 19210 raft_consensus.cc:348] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.941879 19210 raft_consensus.cc:374] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.941960 19210 raft_consensus.cc:729] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.942284 19210 consensus_queue.cc:260] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.942852 19210 ts_tablet_manager.cc:1398] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:18.943219 19210 tablet_bootstrap.cc:492] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78: Bootstrap starting.
I20240502 14:15:18.943709 19218 tablet_bootstrap.cc:654] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.944986 19210 tablet_bootstrap.cc:654] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.948182 19211 tablet_bootstrap.cc:654] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.950898 19211 tablet_bootstrap.cc:492] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f: No bootstrap required, opened a new log
I20240502 14:15:18.951045 19211 ts_tablet_manager.cc:1367] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 14:15:18.951807 19218 tablet_bootstrap.cc:492] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: No bootstrap required, opened a new log
I20240502 14:15:18.951807 19211 raft_consensus.cc:348] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.951979 19218 ts_tablet_manager.cc:1367] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20240502 14:15:18.952059 19211 raft_consensus.cc:374] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.952153 19211 raft_consensus.cc:729] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.952440 19211 consensus_queue.cc:260] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.952749 19218 raft_consensus.cc:348] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.952877 19210 tablet_bootstrap.cc:492] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78: No bootstrap required, opened a new log
I20240502 14:15:18.952967 19218 raft_consensus.cc:374] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.953045 19210 ts_tablet_manager.cc:1367] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 14:15:18.953066 19218 raft_consensus.cc:729] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.953354 19218 consensus_queue.cc:260] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.953763 19210 raft_consensus.cc:348] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.953869 19218 ts_tablet_manager.cc:1398] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:18.953984 19210 raft_consensus.cc:374] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.954092 19210 raft_consensus.cc:729] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.954202 19218 tablet_bootstrap.cc:492] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Bootstrap starting.
I20240502 14:15:18.954403 19210 consensus_queue.cc:260] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.955174 19211 ts_tablet_manager.cc:1398] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:15:18.955611 19210 ts_tablet_manager.cc:1398] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:18.955852 19218 tablet_bootstrap.cc:654] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.955929 19210 tablet_bootstrap.cc:492] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78: Bootstrap starting.
I20240502 14:15:18.957541 19210 tablet_bootstrap.cc:654] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.957671 19211 tablet_bootstrap.cc:492] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f: Bootstrap starting.
I20240502 14:15:18.959734 19211 tablet_bootstrap.cc:654] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.960526 19223 raft_consensus.cc:482] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:18.960716 19223 raft_consensus.cc:504] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.961635 19223 leader_election.cc:288] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:18.962332 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f" is_pre_election: true
I20240502 14:15:18.963346 19165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" is_pre_election: true
W20240502 14:15:18.963516 18982 leader_election.cc:341] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80654f573c764bc18e65d583b751997f (127.15.74.66:44583): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:15:18.963914 18979 leader_election.cc:341] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:18.963994 19218 tablet_bootstrap.cc:492] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: No bootstrap required, opened a new log
I20240502 14:15:18.964043 18979 leader_election.cc:302] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7; no voters: 4c60ef9c4e8f439facdfdc6fd669ae78, 80654f573c764bc18e65d583b751997f
I20240502 14:15:18.964126 19218 ts_tablet_manager.cc:1367] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240502 14:15:18.964529 19223 raft_consensus.cc:2726] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:18.964891 19218 raft_consensus.cc:348] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.965229 19218 raft_consensus.cc:374] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.965375 19218 raft_consensus.cc:729] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.965706 19218 consensus_queue.cc:260] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.966225 19218 ts_tablet_manager.cc:1398] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:18.966362 19211 tablet_bootstrap.cc:492] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f: No bootstrap required, opened a new log
I20240502 14:15:18.966538 19211 ts_tablet_manager.cc:1367] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240502 14:15:18.966624 19218 tablet_bootstrap.cc:492] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Bootstrap starting.
I20240502 14:15:18.967216 19211 raft_consensus.cc:348] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.967520 19211 raft_consensus.cc:374] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.967646 19211 raft_consensus.cc:729] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.967911 19211 consensus_queue.cc:260] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.968155 19218 tablet_bootstrap.cc:654] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.971041 19211 ts_tablet_manager.cc:1398] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:15:18.972311 19211 tablet_bootstrap.cc:492] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f: Bootstrap starting.
I20240502 14:15:18.972625 19210 tablet_bootstrap.cc:492] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78: No bootstrap required, opened a new log
I20240502 14:15:18.972812 19210 ts_tablet_manager.cc:1367] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.010s
I20240502 14:15:18.973585 19210 raft_consensus.cc:348] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.973790 19210 raft_consensus.cc:374] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.973867 19210 raft_consensus.cc:729] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.974048 19210 consensus_queue.cc:260] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.974433 19211 tablet_bootstrap.cc:654] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.974530 19210 ts_tablet_manager.cc:1398] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:18.974856 19210 tablet_bootstrap.cc:492] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78: Bootstrap starting.
I20240502 14:15:18.976589 19210 tablet_bootstrap.cc:654] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.979503 19214 raft_consensus.cc:482] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:18.979633 19218 tablet_bootstrap.cc:492] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: No bootstrap required, opened a new log
I20240502 14:15:18.979737 19218 ts_tablet_manager.cc:1367] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20240502 14:15:18.979661 19214 raft_consensus.cc:504] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.980239 19218 raft_consensus.cc:348] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.980468 19218 raft_consensus.cc:374] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.980573 19218 raft_consensus.cc:729] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.980577 19016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a8747fc61249759eb84a0110df4efb" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" is_pre_election: true
I20240502 14:15:18.980677 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a8747fc61249759eb84a0110df4efb" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f" is_pre_election: true
I20240502 14:15:18.980849 19091 raft_consensus.cc:2454] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 0.
I20240502 14:15:18.980844 19016 raft_consensus.cc:2454] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 0.
I20240502 14:15:18.980845 19218 consensus_queue.cc:260] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.981348 19130 leader_election.cc:302] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78; no voters:
I20240502 14:15:18.980247 19214 leader_election.cc:288] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:18.981350 19218 ts_tablet_manager.cc:1398] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:18.981818 19214 raft_consensus.cc:2781] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:18.981940 19214 raft_consensus.cc:482] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:18.982091 19214 raft_consensus.cc:3037] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.982213 19218 tablet_bootstrap.cc:492] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Bootstrap starting.
I20240502 14:15:18.984535 19218 tablet_bootstrap.cc:654] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.985196 19214 raft_consensus.cc:504] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.985898 19214 leader_election.cc:288] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:18.986325 19016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a8747fc61249759eb84a0110df4efb" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:18.986500 19210 tablet_bootstrap.cc:492] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78: No bootstrap required, opened a new log
I20240502 14:15:18.986510 19016 raft_consensus.cc:3037] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.986579 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a8747fc61249759eb84a0110df4efb" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:18.986680 19210 ts_tablet_manager.cc:1367] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20240502 14:15:18.986768 19091 raft_consensus.cc:3037] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.987004 19211 tablet_bootstrap.cc:492] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f: No bootstrap required, opened a new log
I20240502 14:15:18.987151 19211 ts_tablet_manager.cc:1367] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20240502 14:15:18.987517 19210 raft_consensus.cc:348] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.987747 19210 raft_consensus.cc:374] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.987838 19210 raft_consensus.cc:729] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.988067 19210 consensus_queue.cc:260] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.988622 19210 ts_tablet_manager.cc:1398] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:18.988029 19211 raft_consensus.cc:348] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.988780 19211 raft_consensus.cc:374] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.988854 19211 raft_consensus.cc:729] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.989095 19211 consensus_queue.cc:260] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.989543 19211 ts_tablet_manager.cc:1398] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:18.989688 19016 raft_consensus.cc:2454] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 1.
I20240502 14:15:18.989833 19211 tablet_bootstrap.cc:492] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f: Bootstrap starting.
I20240502 14:15:18.990095 19130 leader_election.cc:302] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78; no voters:
I20240502 14:15:18.989688 19091 raft_consensus.cc:2454] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 1.
I20240502 14:15:18.991488 19214 raft_consensus.cc:2781] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:18.991572 19211 tablet_bootstrap.cc:654] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.991706 19214 raft_consensus.cc:686] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Becoming Leader. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Running, Role: LEADER
I20240502 14:15:18.992085 19214 consensus_queue.cc:237] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.988881 19210 tablet_bootstrap.cc:492] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78: Bootstrap starting.
I20240502 14:15:18.993592 19218 tablet_bootstrap.cc:492] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: No bootstrap required, opened a new log
I20240502 14:15:18.993790 19218 ts_tablet_manager.cc:1367] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20240502 14:15:18.994627 19218 raft_consensus.cc:348] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.994890 19218 raft_consensus.cc:374] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.995088 19218 raft_consensus.cc:729] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.995384 19218 consensus_queue.cc:260] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.995941 19218 ts_tablet_manager.cc:1398] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:18.996374 19218 tablet_bootstrap.cc:492] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Bootstrap starting.
I20240502 14:15:18.996482 19210 tablet_bootstrap.cc:654] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.998298 19218 tablet_bootstrap.cc:654] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.998661 19211 tablet_bootstrap.cc:492] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f: No bootstrap required, opened a new log
I20240502 14:15:18.998807 19211 ts_tablet_manager.cc:1367] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20240502 14:15:18.999547 19211 raft_consensus.cc:348] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:18.999773 19211 raft_consensus.cc:374] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.999861 19211 raft_consensus.cc:729] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.000109 19211 consensus_queue.cc:260] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.000623 19211 ts_tablet_manager.cc:1398] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:19.004175 18912 catalog_manager.cc:5484] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67). New cstate: current_term: 1 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:19.004880 19211 tablet_bootstrap.cc:492] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f: Bootstrap starting.
I20240502 14:15:19.006337 19211 tablet_bootstrap.cc:654] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:19.007539 19210 tablet_bootstrap.cc:492] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78: No bootstrap required, opened a new log
I20240502 14:15:19.007740 19210 ts_tablet_manager.cc:1367] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.003s
I20240502 14:15:19.008148 19218 tablet_bootstrap.cc:492] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: No bootstrap required, opened a new log
I20240502 14:15:19.008292 19218 ts_tablet_manager.cc:1367] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240502 14:15:19.008608 19211 tablet_bootstrap.cc:492] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f: No bootstrap required, opened a new log
I20240502 14:15:19.008530 19210 raft_consensus.cc:348] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.008736 19210 raft_consensus.cc:374] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:19.008778 19211 ts_tablet_manager.cc:1367] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:15:19.008828 19210 raft_consensus.cc:729] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.008976 19218 raft_consensus.cc:348] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.009061 19210 consensus_queue.cc:260] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.009361 19211 raft_consensus.cc:348] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.009150 19218 raft_consensus.cc:374] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:19.009594 19211 raft_consensus.cc:374] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:19.009694 19218 raft_consensus.cc:729] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.009593 19210 ts_tablet_manager.cc:1398] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:19.009943 19218 consensus_queue.cc:260] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.010531 19218 ts_tablet_manager.cc:1398] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:19.010844 19218 tablet_bootstrap.cc:492] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Bootstrap starting.
I20240502 14:15:19.009717 19211 raft_consensus.cc:729] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.011750 19211 consensus_queue.cc:260] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.012287 19211 ts_tablet_manager.cc:1398] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:15:19.012434 19218 tablet_bootstrap.cc:654] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:19.014125 19218 tablet_bootstrap.cc:492] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: No bootstrap required, opened a new log
I20240502 14:15:19.014254 19218 ts_tablet_manager.cc:1367] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 14:15:19.014796 19218 raft_consensus.cc:348] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.014991 19218 raft_consensus.cc:374] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:19.015092 19218 raft_consensus.cc:729] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.015308 19218 consensus_queue.cc:260] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.015575 19253 raft_consensus.cc:482] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.015799 19218 ts_tablet_manager.cc:1398] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:15:19.015767 19253 raft_consensus.cc:504] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.016824 19016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342705e9da8043ae9aebe50310650d2c" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" is_pre_election: true
I20240502 14:15:19.016976 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342705e9da8043ae9aebe50310650d2c" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f" is_pre_election: true
I20240502 14:15:19.017017 19016 raft_consensus.cc:2454] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 0.
I20240502 14:15:19.017158 19091 raft_consensus.cc:2454] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 0.
I20240502 14:15:19.017379 19130 leader_election.cc:302] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78; no voters:
I20240502 14:15:19.017599 19253 leader_election.cc:288] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:19.017652 19214 raft_consensus.cc:2781] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:19.017732 19214 raft_consensus.cc:482] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:19.017813 19214 raft_consensus.cc:3037] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.019905 19214 raft_consensus.cc:504] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.020349 19214 leader_election.cc:288] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:19.020641 19016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342705e9da8043ae9aebe50310650d2c" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:19.020736 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342705e9da8043ae9aebe50310650d2c" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:19.020795 19016 raft_consensus.cc:3037] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.020915 19091 raft_consensus.cc:3037] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.022715 19091 raft_consensus.cc:2454] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 1.
I20240502 14:15:19.022719 19016 raft_consensus.cc:2454] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 1.
I20240502 14:15:19.023088 19131 leader_election.cc:302] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:19.023401 19214 raft_consensus.cc:2781] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:19.023559 19214 raft_consensus.cc:686] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Becoming Leader. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Running, Role: LEADER
I20240502 14:15:19.023784 19214 consensus_queue.cc:237] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.025897 18913 catalog_manager.cc:5484] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67). New cstate: current_term: 1 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:19.034626 19215 raft_consensus.cc:482] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.034749 19215 raft_consensus.cc:504] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.035452 19215 leader_election.cc:288] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.041554 19214 raft_consensus.cc:482] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.041734 19214 raft_consensus.cc:504] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.041891 19016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" is_pre_election: true
I20240502 14:15:19.041891 19165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" is_pre_election: true
I20240502 14:15:19.042171 19165 raft_consensus.cc:2454] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 0.
I20240502 14:15:19.042174 19016 raft_consensus.cc:2454] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 0.
I20240502 14:15:19.042407 19214 leader_election.cc:288] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:19.042564 19056 leader_election.cc:302] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:19.042867 19215 raft_consensus.cc:2781] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:19.042867 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f" is_pre_election: true
I20240502 14:15:19.043015 19215 raft_consensus.cc:482] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:19.042929 19015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" is_pre_election: true
I20240502 14:15:19.043114 19215 raft_consensus.cc:3037] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.043207 19015 raft_consensus.cc:2454] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 0.
I20240502 14:15:19.043535 19130 leader_election.cc:302] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78; no voters:
I20240502 14:15:19.043746 19214 raft_consensus.cc:2781] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:19.043823 19214 raft_consensus.cc:482] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:19.043889 19214 raft_consensus.cc:3037] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.045125 19215 raft_consensus.cc:504] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.045405 19091 raft_consensus.cc:2379] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in current term 1: Already voted for candidate 80654f573c764bc18e65d583b751997f in this term.
I20240502 14:15:19.045739 19215 leader_election.cc:288] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.045728 19214 raft_consensus.cc:504] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.046164 19015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:19.046355 19015 raft_consensus.cc:3037] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.046579 19016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:19.046578 19165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:19.046861 19165 raft_consensus.cc:2379] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 80654f573c764bc18e65d583b751997f in current term 1: Already voted for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in this term.
I20240502 14:15:19.047406 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:19.047654 19091 raft_consensus.cc:2379] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in current term 1: Already voted for candidate 80654f573c764bc18e65d583b751997f in this term.
I20240502 14:15:19.047853 19214 leader_election.cc:288] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:19.048048 19131 leader_election.cc:302] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c60ef9c4e8f439facdfdc6fd669ae78; no voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f
I20240502 14:15:19.048293 19214 raft_consensus.cc:2726] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:15:19.049048 19015 raft_consensus.cc:2454] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 1.
I20240502 14:15:19.049345 19056 leader_election.cc:302] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters: 4c60ef9c4e8f439facdfdc6fd669ae78
I20240502 14:15:19.049679 19215 raft_consensus.cc:2781] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:19.050076 19215 raft_consensus.cc:686] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Becoming Leader. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:19.050334 19215 consensus_queue.cc:237] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.052760 18913 catalog_manager.cc:5484] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80654f573c764bc18e65d583b751997f (127.15.74.66). New cstate: current_term: 1 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:19.060254 19220 raft_consensus.cc:482] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.060384 19220 raft_consensus.cc:504] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.060873 19220 leader_election.cc:288] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.061218 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f" is_pre_election: true
I20240502 14:15:19.061287 19165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" is_pre_election: true
I20240502 14:15:19.061388 19091 raft_consensus.cc:2454] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 0.
I20240502 14:15:19.061477 19165 raft_consensus.cc:2454] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 0.
I20240502 14:15:19.061709 18982 leader_election.cc:302] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:19.061937 19220 raft_consensus.cc:2781] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:19.062069 19220 raft_consensus.cc:482] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:19.062167 19220 raft_consensus.cc:3037] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.064044 19220 raft_consensus.cc:504] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.064517 19220 leader_election.cc:288] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 election: Requested vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.064899 19165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:19.064916 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:19.065069 19165 raft_consensus.cc:3037] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.065083 19091 raft_consensus.cc:3037] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.066030 19215 raft_consensus.cc:482] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.066181 19215 raft_consensus.cc:504] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.067024 19015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99128e8f96dc4e1582ec51cbcb8085e0" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" is_pre_election: true
I20240502 14:15:19.067215 19015 raft_consensus.cc:2454] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 0.
I20240502 14:15:19.067127 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99128e8f96dc4e1582ec51cbcb8085e0" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" is_pre_election: true
I20240502 14:15:19.067340 19215 leader_election.cc:288] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.067399 19164 raft_consensus.cc:2454] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 0.
I20240502 14:15:19.067471 19165 raft_consensus.cc:2454] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 1.
I20240502 14:15:19.067605 19091 raft_consensus.cc:2454] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 1.
I20240502 14:15:19.067826 18979 leader_election.cc:302] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78; no voters:
I20240502 14:15:19.067951 19056 leader_election.cc:302] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:19.068101 19220 raft_consensus.cc:2781] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:19.068243 19220 raft_consensus.cc:686] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Becoming Leader. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:19.068413 19215 raft_consensus.cc:2781] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:19.068558 19215 raft_consensus.cc:482] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:19.068503 19220 consensus_queue.cc:237] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.068681 19215 raft_consensus.cc:3037] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.070966 19215 raft_consensus.cc:504] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.070896 18913 catalog_manager.cc:5484] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65). New cstate: current_term: 1 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:19.071480 19215 leader_election.cc:288] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.071839 19015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99128e8f96dc4e1582ec51cbcb8085e0" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:19.071949 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99128e8f96dc4e1582ec51cbcb8085e0" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:19.072064 19015 raft_consensus.cc:3037] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.072149 19164 raft_consensus.cc:3037] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.074550 19015 raft_consensus.cc:2454] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 1.
I20240502 14:15:19.074895 19056 leader_election.cc:302] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:19.074977 19164 raft_consensus.cc:2454] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 1.
I20240502 14:15:19.075196 19215 raft_consensus.cc:2781] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:19.075348 19215 raft_consensus.cc:686] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Becoming Leader. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:19.075614 19215 consensus_queue.cc:237] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.077814 18913 catalog_manager.cc:5484] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80654f573c764bc18e65d583b751997f (127.15.74.66). New cstate: current_term: 1 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:19.084674 19215 raft_consensus.cc:482] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.084822 19215 raft_consensus.cc:504] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.085301 19215 leader_election.cc:288] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.085690 19015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" is_pre_election: true
I20240502 14:15:19.085747 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" is_pre_election: true
I20240502 14:15:19.086015 19164 raft_consensus.cc:2379] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 80654f573c764bc18e65d583b751997f in current term 1: Already voted for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in this term.
I20240502 14:15:19.086354 19054 leader_election.cc:302] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80654f573c764bc18e65d583b751997f; no voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78
I20240502 14:15:19.086598 19215 raft_consensus.cc:3037] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.088958 19215 raft_consensus.cc:2726] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:19.118491 19215 raft_consensus.cc:482] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.118696 19215 raft_consensus.cc:504] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.119335 19215 leader_election.cc:288] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.119748 19015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c203a4764245dba0944537c7d10129" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" is_pre_election: true
I20240502 14:15:19.119823 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c203a4764245dba0944537c7d10129" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" is_pre_election: true
I20240502 14:15:19.119927 19015 raft_consensus.cc:2454] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 0.
I20240502 14:15:19.120044 19164 raft_consensus.cc:2454] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 0.
I20240502 14:15:19.120293 19056 leader_election.cc:302] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:19.120565 19215 raft_consensus.cc:2781] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:19.120651 19215 raft_consensus.cc:482] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:19.120728 19215 raft_consensus.cc:3037] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.122421 19215 raft_consensus.cc:504] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.122995 19215 leader_election.cc:288] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.123298 19015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c203a4764245dba0944537c7d10129" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:19.123404 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c203a4764245dba0944537c7d10129" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:19.123545 19015 raft_consensus.cc:3037] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.123621 19164 raft_consensus.cc:3037] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.126319 19015 raft_consensus.cc:2454] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 1.
I20240502 14:15:19.126320 19164 raft_consensus.cc:2454] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 1.
I20240502 14:15:19.126690 19056 leader_election.cc:302] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:19.126891 19215 raft_consensus.cc:2781] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:19.127043 19215 raft_consensus.cc:686] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Becoming Leader. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:19.127297 19215 consensus_queue.cc:237] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.129693 18913 catalog_manager.cc:5484] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80654f573c764bc18e65d583b751997f (127.15.74.66). New cstate: current_term: 1 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:19.132931 19214 raft_consensus.cc:482] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.133054 19214 raft_consensus.cc:504] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.133611 19214 leader_election.cc:288] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141)
I20240502 14:15:19.133977 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f" is_pre_election: true
I20240502 14:15:19.134019 19015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" is_pre_election: true
I20240502 14:15:19.134167 19091 raft_consensus.cc:2454] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 0.
I20240502 14:15:19.134171 19015 raft_consensus.cc:2454] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 0.
I20240502 14:15:19.134479 19131 leader_election.cc:302] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c60ef9c4e8f439facdfdc6fd669ae78, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:19.134708 19214 raft_consensus.cc:2781] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:19.134816 19214 raft_consensus.cc:482] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:19.134884 19214 raft_consensus.cc:3037] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.136600 19214 raft_consensus.cc:504] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.137127 19214 leader_election.cc:288] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 election: Requested vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141)
I20240502 14:15:19.137419 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:19.137526 19015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:19.137612 19091 raft_consensus.cc:3037] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.137701 19015 raft_consensus.cc:3037] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.139470 19091 raft_consensus.cc:2454] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 1.
I20240502 14:15:19.139523 19015 raft_consensus.cc:2454] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 1.
I20240502 14:15:19.139876 19131 leader_election.cc:302] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:19.140129 19214 raft_consensus.cc:2781] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:19.140276 19214 raft_consensus.cc:686] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Becoming Leader. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Running, Role: LEADER
I20240502 14:15:19.140570 19214 consensus_queue.cc:237] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.142871 18913 catalog_manager.cc:5484] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67). New cstate: current_term: 1 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:19.150988 19220 raft_consensus.cc:482] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.151176 19220 raft_consensus.cc:504] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.151820 19220 leader_election.cc:288] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.152268 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f465cbf06f4420c8642e8092ce8e1ed" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" is_pre_election: true
I20240502 14:15:19.152331 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f465cbf06f4420c8642e8092ce8e1ed" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f" is_pre_election: true
I20240502 14:15:19.152473 19164 raft_consensus.cc:2454] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 0.
I20240502 14:15:19.152514 19091 raft_consensus.cc:2454] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 0.
I20240502 14:15:19.152832 18979 leader_election.cc:302] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78; no voters:
I20240502 14:15:19.153098 19220 raft_consensus.cc:2781] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:19.153224 19220 raft_consensus.cc:482] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:19.153340 19220 raft_consensus.cc:3037] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.155339 19220 raft_consensus.cc:504] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.155925 19220 leader_election.cc:288] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 election: Requested vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.156379 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f465cbf06f4420c8642e8092ce8e1ed" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:19.156379 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f465cbf06f4420c8642e8092ce8e1ed" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:19.156554 19091 raft_consensus.cc:3037] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.156591 19164 raft_consensus.cc:3037] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.158399 19091 raft_consensus.cc:2454] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 1.
I20240502 14:15:19.158622 19164 raft_consensus.cc:2454] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 1.
I20240502 14:15:19.158788 18982 leader_election.cc:302] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:19.159094 19220 raft_consensus.cc:2781] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:19.159219 19220 raft_consensus.cc:686] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Becoming Leader. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:19.159524 19220 consensus_queue.cc:237] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.161924 18913 catalog_manager.cc:5484] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65). New cstate: current_term: 1 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:19.175309 19220 raft_consensus.cc:482] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.175558 19220 raft_consensus.cc:504] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.176110 19220 leader_election.cc:288] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.176597 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2dcb2456949cfa34b5c2e8b0d2e6d" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" is_pre_election: true
I20240502 14:15:19.176594 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2dcb2456949cfa34b5c2e8b0d2e6d" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f" is_pre_election: true
I20240502 14:15:19.176812 19164 raft_consensus.cc:2454] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 0.
I20240502 14:15:19.176862 19091 raft_consensus.cc:2454] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 0.
I20240502 14:15:19.177174 18979 leader_election.cc:302] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78; no voters:
I20240502 14:15:19.177459 19220 raft_consensus.cc:2781] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:19.177539 19220 raft_consensus.cc:482] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:19.177625 19220 raft_consensus.cc:3037] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.179422 19220 raft_consensus.cc:504] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.180006 19220 leader_election.cc:288] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 election: Requested vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.180302 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2dcb2456949cfa34b5c2e8b0d2e6d" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:19.180404 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2dcb2456949cfa34b5c2e8b0d2e6d" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:19.180495 19091 raft_consensus.cc:3037] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.180608 19164 raft_consensus.cc:3037] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.182467 19164 raft_consensus.cc:2454] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 1.
I20240502 14:15:19.182484 19091 raft_consensus.cc:2454] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 1.
I20240502 14:15:19.182845 18979 leader_election.cc:302] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78; no voters:
I20240502 14:15:19.183107 19220 raft_consensus.cc:2781] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:19.183284 19220 raft_consensus.cc:686] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Becoming Leader. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:19.183573 19220 consensus_queue.cc:237] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.186069 18913 catalog_manager.cc:5484] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65). New cstate: current_term: 1 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:19.228854 19215 raft_consensus.cc:482] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.229059 19215 raft_consensus.cc:504] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.229846 19215 leader_election.cc:288] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.230350 19015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f260d59158a344beb686ea0f9746f743" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" is_pre_election: true
I20240502 14:15:19.230398 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f260d59158a344beb686ea0f9746f743" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" is_pre_election: true
I20240502 14:15:19.230587 19015 raft_consensus.cc:2454] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 0.
I20240502 14:15:19.230659 19164 raft_consensus.cc:2454] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 0.
I20240502 14:15:19.230966 19056 leader_election.cc:302] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:19.231251 19215 raft_consensus.cc:2781] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:19.231365 19215 raft_consensus.cc:482] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:19.231467 19215 raft_consensus.cc:3037] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.233351 19215 raft_consensus.cc:504] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.233887 19215 leader_election.cc:288] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.234287 19015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f260d59158a344beb686ea0f9746f743" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:19.234329 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f260d59158a344beb686ea0f9746f743" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:19.234457 19015 raft_consensus.cc:3037] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.234541 19164 raft_consensus.cc:3037] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.236410 19015 raft_consensus.cc:2454] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 1.
I20240502 14:15:19.236719 19164 raft_consensus.cc:2454] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 1.
I20240502 14:15:19.236806 19056 leader_election.cc:302] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:19.237107 19215 raft_consensus.cc:2781] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:19.237260 19215 raft_consensus.cc:686] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Becoming Leader. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:19.237589 19215 consensus_queue.cc:237] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.247182 18913 catalog_manager.cc:5484] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80654f573c764bc18e65d583b751997f (127.15.74.66). New cstate: current_term: 1 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:19.296924 19220 raft_consensus.cc:482] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.297124 19220 raft_consensus.cc:504] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.297804 19220 leader_election.cc:288] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.298326 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618d11b0b08e4d49ac15d443acd10e2c" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" is_pre_election: true
I20240502 14:15:19.298297 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618d11b0b08e4d49ac15d443acd10e2c" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f" is_pre_election: true
I20240502 14:15:19.298545 19164 raft_consensus.cc:2454] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 0.
I20240502 14:15:19.298543 19091 raft_consensus.cc:2454] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 0.
I20240502 14:15:19.298931 18982 leader_election.cc:302] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:19.299156 19220 raft_consensus.cc:2781] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:19.299233 19220 raft_consensus.cc:482] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:19.299305 19220 raft_consensus.cc:3037] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.301074 19220 raft_consensus.cc:504] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.301585 19220 leader_election.cc:288] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 election: Requested vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:19.301951 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618d11b0b08e4d49ac15d443acd10e2c" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:19.302048 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618d11b0b08e4d49ac15d443acd10e2c" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:19.302184 19164 raft_consensus.cc:3037] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.302184 19091 raft_consensus.cc:3037] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.304848 19091 raft_consensus.cc:2454] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 1.
I20240502 14:15:19.304848 19164 raft_consensus.cc:2454] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 1.
I20240502 14:15:19.305249 18979 leader_election.cc:302] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78; no voters:
I20240502 14:15:19.305460 19220 raft_consensus.cc:2781] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:19.305608 19220 raft_consensus.cc:686] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Becoming Leader. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:19.305838 19220 consensus_queue.cc:237] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.308413 18913 catalog_manager.cc:5484] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65). New cstate: current_term: 1 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:19.343091 19220 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.363682 19283 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.388365 19164 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.388448 19091 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.389043 19282 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.389230 19281 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.390206 19091 raft_consensus.cc:1261] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.392691 19164 raft_consensus.cc:1261] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.392740 19220 consensus_queue.cc:1035] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.394913 19090 raft_consensus.cc:1261] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.395294 19214 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.397812 18967 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:19.398033 19296 consensus_queue.cc:1035] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.399796 19089 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:19.400307 19163 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.400369 19283 consensus_queue.cc:1035] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.401108 19281 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.400300 19162 raft_consensus.cc:1261] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.401613 19281 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.407336 19283 consensus_queue.cc:1035] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.439400 19253 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.456265 19016 raft_consensus.cc:1261] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.457044 19253 consensus_queue.cc:1035] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.457204 19085 raft_consensus.cc:1261] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.470762 19214 consensus_queue.cc:1035] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.490657 19084 raft_consensus.cc:1261] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.491199 19014 raft_consensus.cc:1261] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.491820 19253 consensus_queue.cc:1035] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.491921 19014 raft_consensus.cc:1261] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.492075 19214 consensus_queue.cc:1035] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.492681 19215 consensus_queue.cc:1035] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.496224 19165 raft_consensus.cc:1261] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.497236 19215 consensus_queue.cc:1035] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.560109 19011 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.560891 19158 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.606567 19215 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.046s
I20240502 14:15:19.619378 19012 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
mode: GRACEFUL
new_leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
from {username='slave'} at 127.0.0.1:54318
W20240502 14:15:19.619613 19307 log.cc:924] Time spent T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f: Append to log took a long time: real 0.060s user 0.000s sys 0.003s
I20240502 14:15:19.619657 19012 raft_consensus.cc:595] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Received request to transfer leadership to TS 4c60ef9c4e8f439facdfdc6fd669ae78
I20240502 14:15:19.620466 18967 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 14:15:19.621024 19159 raft_consensus.cc:1261] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.622951 18967 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:19.648128 19012 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.648931 19215 consensus_queue.cc:1035] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.653836 19215 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.666630 19290 mvcc.cc:205] Tried to move back new op lower bound from 7023244572531937280 to 7023244571201740800. Current Snapshot: MvccSnapshot[applied={T|T < 7023244572478111744}]
W20240502 14:15:19.690433 19294 log.cc:924] Time spent T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20240502 14:15:19.717734 19320 log.cc:924] Time spent T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20240502 14:15:19.721491 19223 raft_consensus.cc:979] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: : Instructing follower 4c60ef9c4e8f439facdfdc6fd669ae78 to start an election
I20240502 14:15:19.721604 19223 raft_consensus.cc:1067] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Signalling peer 4c60ef9c4e8f439facdfdc6fd669ae78 to start an election
W20240502 14:15:19.646512 19279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54310: Service unavailable: leader transfer in progress
W20240502 14:15:19.762095 19332 log.cc:924] Time spent T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20240502 14:15:19.763556 19160 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1"
dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
from {username='slave'} at 127.0.0.1:49454
I20240502 14:15:19.763742 19160 raft_consensus.cc:482] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:19.763834 19160 raft_consensus.cc:3037] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:15:19.772877 19300 log.cc:924] Time spent T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
W20240502 14:15:19.802923 19054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.74.67:39767, user_credentials={real_user=slave}} blocked reactor thread for 227075us
I20240502 14:15:19.803947 19164 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:19.804601 18995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54310: Service unavailable: leader transfer in progress
I20240502 14:15:19.806773 19329 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.806866 19264 consensus_queue.cc:1035] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.807384 19329 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:15:19.813050 18995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54310: Service unavailable: leader transfer in progress
W20240502 14:15:19.826124 18995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54310: Service unavailable: leader transfer in progress
W20240502 14:15:19.829289 19331 log.cc:924] Time spent T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78: Append to log took a long time: real 0.183s user 0.000s sys 0.001s
W20240502 14:15:19.869017 18995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54310: Service unavailable: leader transfer in progress
I20240502 14:15:19.881058 19160 raft_consensus.cc:504] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.881898 19160 leader_election.cc:288] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 2 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:19.882229 19011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:19.882613 19084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:19.883783 19011 raft_consensus.cc:3032] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:19.883872 19011 raft_consensus.cc:729] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
W20240502 14:15:19.888500 18995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54310: Illegal state: replica 1880cd4d2b8e4ec6b42dfd8d2660f5d7 is not leader of this config: current role FOLLOWER
I20240502 14:15:19.888096 19011 consensus_queue.cc:260] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.891492 19011 raft_consensus.cc:3037] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:15:19.904688 19069 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32780: Illegal state: replica 80654f573c764bc18e65d583b751997f is not leader of this config: current role FOLLOWER
I20240502 14:15:19.911017 19291 mvcc.cc:205] Tried to move back new op lower bound from 7023244573180350464 to 7023244570832646144. Current Snapshot: MvccSnapshot[applied={T|T < 7023244572846129152 or (T in {7023244573180350464,7023244573133746176,7023244573120823296})}]
I20240502 14:15:19.913504 19011 raft_consensus.cc:2454] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 2.
I20240502 14:15:19.916440 19130 leader_election.cc:302] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78; no voters: 80654f573c764bc18e65d583b751997f
I20240502 14:15:19.916687 19341 raft_consensus.cc:2781] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:19.916807 19341 raft_consensus.cc:686] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Becoming Leader. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Running, Role: LEADER
I20240502 14:15:19.918604 19341 consensus_queue.cc:237] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:19.939136 19013 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:15:19.941200 19253 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:19.944088 19084 raft_consensus.cc:3037] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:19.970659 19084 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:15:19.977138 19316 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:19.984679 18911 catalog_manager.cc:5484] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: term changed from 1 to 2, leader changed from 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) to 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67). New cstate: current_term: 2 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:20.799415 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:20.806958 19352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:20.807747 19353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:20.812897 19355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:20.813489 15657 server_base.cc:1034] running on GCE node
I20240502 14:15:20.813696 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:15:20.813745 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:15:20.813771 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659320813768 us; error 0 us; skew 500 ppm
I20240502 14:15:20.813871 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:20.815385 15657 webserver.cc:460] Webserver started at http://127.15.74.68:46143/ using document root <none> and password file <none>
I20240502 14:15:20.815634 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:20.815696 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:20.815786 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:20.816375 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-3-root/instance:
uuid: "a2f6fcadc2e047ddb0c111c95420861a"
format_stamp: "Formatted at 2024-05-02 14:15:20 on dist-test-slave-wbm6"
I20240502 14:15:20.818989 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 14:15:20.820556 19360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:20.820876 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:15:20.820967 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-3-root
uuid: "a2f6fcadc2e047ddb0c111c95420861a"
format_stamp: "Formatted at 2024-05-02 14:15:20 on dist-test-slave-wbm6"
I20240502 14:15:20.821076 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:20.841925 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:20.842489 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:20.843060 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:20.843982 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:20.844033 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:20.844090 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:20.844139 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:20.860273 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.68:41443
I20240502 14:15:20.860309 19423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.68:41443 every 8 connection(s)
I20240502 14:15:20.862098 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:20.865839 19428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:20.866438 19429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:20.866883 19431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:20.867627 15657 server_base.cc:1034] running on GCE node
I20240502 14:15:20.867859 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:15:20.867915 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:15:20.868434 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659320868428 us; error 0 us; skew 500 ppm
I20240502 14:15:20.868592 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:20.874894 15657 webserver.cc:460] Webserver started at http://127.15.74.69:39211/ using document root <none> and password file <none>
I20240502 14:15:20.875161 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:20.875242 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:20.875391 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:20.875416 19424 heartbeater.cc:347] Connected to a master server at 127.15.74.126:42275
I20240502 14:15:20.875571 19424 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:20.876022 19424 heartbeater.cc:511] Master 127.15.74.126:42275 requested a full tablet report, sending...
I20240502 14:15:20.876076 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-4-root/instance:
uuid: "eac995eafe3e4c388a7caf591d6f1c66"
format_stamp: "Formatted at 2024-05-02 14:15:20 on dist-test-slave-wbm6"
I20240502 14:15:20.876955 18905 ts_manager.cc:194] Registered new tserver with Master: a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68:41443)
I20240502 14:15:20.877758 18905 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51758
I20240502 14:15:20.878990 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240502 14:15:20.880571 19436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:20.880977 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:15:20.881072 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-4-root
uuid: "eac995eafe3e4c388a7caf591d6f1c66"
format_stamp: "Formatted at 2024-05-02 14:15:20 on dist-test-slave-wbm6"
I20240502 14:15:20.881184 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:20.886608 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:20.887079 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:20.887580 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:20.888381 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:20.888448 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:20.888528 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:20.888574 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:20.903352 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.69:44877
I20240502 14:15:20.903383 19499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.69:44877 every 8 connection(s)
I20240502 14:15:20.905097 15657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:20.908177 19503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:20.908916 19504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:20.911708 19506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:20.912189 19500 heartbeater.cc:347] Connected to a master server at 127.15.74.126:42275
I20240502 14:15:20.912353 15657 server_base.cc:1034] running on GCE node
I20240502 14:15:20.912410 19500 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:20.912626 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:15:20.912716 15657 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:15:20.912786 15657 hybrid_clock.cc:648] HybridClock initialized: now 1714659320912761 us; error 0 us; skew 500 ppm
I20240502 14:15:20.912868 19500 heartbeater.cc:511] Master 127.15.74.126:42275 requested a full tablet report, sending...
I20240502 14:15:20.912968 15657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:20.913836 18905 ts_manager.cc:194] Registered new tserver with Master: eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69:44877)
I20240502 14:15:20.914431 15657 webserver.cc:460] Webserver started at http://127.15.74.70:33709/ using document root <none> and password file <none>
I20240502 14:15:20.914705 15657 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:20.914803 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:20.914758 18905 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51760
I20240502 14:15:20.914984 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:20.915649 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-5-root/instance:
uuid: "d908205271af490dada0a2845191fef4"
format_stamp: "Formatted at 2024-05-02 14:15:20 on dist-test-slave-wbm6"
I20240502 14:15:20.918121 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:20.919677 19511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:20.920023 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:15:20.920152 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-5-root
uuid: "d908205271af490dada0a2845191fef4"
format_stamp: "Formatted at 2024-05-02 14:15:20 on dist-test-slave-wbm6"
I20240502 14:15:20.920262 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:20.956754 15657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:20.957274 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:20.957811 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:20.958614 15657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:20.958669 15657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:20.958739 15657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:20.958775 15657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:20.971716 15657 rpc_server.cc:307] RPC server started. Bound to: 127.15.74.70:35089
I20240502 14:15:20.971768 19573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.74.70:35089 every 8 connection(s)
I20240502 14:15:20.977846 19574 heartbeater.cc:347] Connected to a master server at 127.15.74.126:42275
I20240502 14:15:20.977979 19574 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:20.978302 19574 heartbeater.cc:511] Master 127.15.74.126:42275 requested a full tablet report, sending...
I20240502 14:15:20.979084 18905 ts_manager.cc:194] Registered new tserver with Master: d908205271af490dada0a2845191fef4 (127.15.74.70:35089)
I20240502 14:15:20.979861 18905 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51772
I20240502 14:15:21.425760 19011 consensus_queue.cc:237] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:21.427894 19088 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 14:15:21.428392 19160 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 14:15:21.428782 19579 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:21.429108 19296 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:21.431753 19582 raft_consensus.cc:2932] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
I20240502 14:15:21.432096 19088 raft_consensus.cc:2932] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
I20240502 14:15:21.432309 19160 raft_consensus.cc:2932] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
W20240502 14:15:21.436148 18982 consensus_peers.cc:485] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 -> Peer eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69:44877): Couldn't send request to peer eac995eafe3e4c388a7caf591d6f1c66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f465cbf06f4420c8642e8092ce8e1ed. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:21.438141 18905 catalog_manager.cc:5484] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: config changed from index -1 to 26, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New cstate: current_term: 1 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
I20240502 14:15:21.444929 19088 consensus_queue.cc:237] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:21.447119 19011 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20240502 14:15:21.447428 19160 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20240502 14:15:21.447715 19215 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240502 14:15:21.447995 19588 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240502 14:15:21.450265 19215 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index -1 to 24, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:21.451546 19011 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index -1 to 24, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:21.453162 19160 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index -1 to 24, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
W20240502 14:15:21.453976 19057 consensus_peers.cc:485] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f -> Peer a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68:41443): Couldn't send request to peer a2f6fcadc2e047ddb0c111c95420861a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99128e8f96dc4e1582ec51cbcb8085e0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:21.457185 18911 catalog_manager.cc:5484] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: config changed from index -1 to 24, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New cstate: current_term: 1 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:21.468174 19160 consensus_queue.cc:237] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:21.470466 19011 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 14:15:21.470615 19088 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 14:15:21.471007 19593 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:21.471288 19351 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:21.473356 19351 raft_consensus.cc:2932] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:21.475863 19011 raft_consensus.cc:2932] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:21.475965 19088 raft_consensus.cc:2932] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:21.478834 19160 consensus_queue.cc:237] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:21.479003 18905 catalog_manager.cc:5484] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: config changed from index -1 to 26, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New cstate: current_term: 2 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:15:21.481117 19131 consensus_peers.cc:485] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 -> Peer d908205271af490dada0a2845191fef4 (127.15.74.70:35089): Couldn't send request to peer d908205271af490dada0a2845191fef4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35d01415abc44ec4b52577992892c5b1. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:21.481132 19012 raft_consensus.cc:1261] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 14:15:21.481806 19593 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
W20240502 14:15:21.482075 19131 consensus_peers.cc:485] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 -> Peer d908205271af490dada0a2845191fef4 (127.15.74.70:35089): Couldn't send request to peer d908205271af490dada0a2845191fef4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76a8747fc61249759eb84a0110df4efb. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:21.484198 19593 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:21.484630 19012 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:21.488309 19088 raft_consensus.cc:1261] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 14:15:21.489111 19597 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:21.490172 18905 catalog_manager.cc:5484] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: config changed from index -1 to 26, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New cstate: current_term: 1 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:21.490371 19088 consensus_queue.cc:237] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:21.490747 19090 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:21.493108 19160 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 14:15:21.493597 19215 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 14:15:21.495914 19591 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
I20240502 14:15:21.496556 19158 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
W20240502 14:15:21.498955 19057 consensus_peers.cc:485] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f -> Peer eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69:44877): Couldn't send request to peer eac995eafe3e4c388a7caf591d6f1c66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73c203a4764245dba0944537c7d10129. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:21.501163 18905 catalog_manager.cc:5484] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f reported cstate change: config changed from index -1 to 25, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New cstate: current_term: 1 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:21.502111 19012 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 14:15:21.502290 19011 consensus_queue.cc:237] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:21.504366 19215 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 14:15:21.506093 19160 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 14:15:21.506235 19090 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 14:15:21.506546 19296 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:21.507164 19296 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:21.507634 19012 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
I20240502 14:15:21.509583 19296 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:21.510017 19160 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
W20240502 14:15:21.512979 18982 consensus_peers.cc:485] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 -> Peer a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68:41443): Couldn't send request to peer a2f6fcadc2e047ddb0c111c95420861a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 567ee47202bd46048e318b4dbbf3ff03. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:21.513244 19090 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:21.516480 18911 catalog_manager.cc:5484] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: config changed from index -1 to 26, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New cstate: current_term: 1 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:21.623261 18967 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:21.632059 19160 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d"
dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
mode: GRACEFUL
new_leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
from {username='slave'} at 127.0.0.1:49498
I20240502 14:15:21.632259 19160 raft_consensus.cc:595] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Received request to transfer leadership to TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7
I20240502 14:15:21.633183 19012 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
mode: GRACEFUL
new_leader_uuid: "80654f573c764bc18e65d583b751997f"
from {username='slave'} at 127.0.0.1:54318
I20240502 14:15:21.633359 19012 raft_consensus.cc:595] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Received request to transfer leadership to TS 80654f573c764bc18e65d583b751997f
I20240502 14:15:21.633780 18967 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 14:15:21.633976 18967 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:21.782915 19596 raft_consensus.cc:979] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78: : Instructing follower 1880cd4d2b8e4ec6b42dfd8d2660f5d7 to start an election
I20240502 14:15:21.783058 19596 raft_consensus.cc:1067] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Signalling peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7 to start an election
I20240502 14:15:21.783715 19012 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
from {username='slave'} at 127.0.0.1:54284
I20240502 14:15:21.783921 19012 raft_consensus.cc:482] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:21.784018 19012 raft_consensus.cc:3037] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:21.786459 19012 raft_consensus.cc:504] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:21.787011 19012 leader_election.cc:288] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 2 election: Requested vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:21.787379 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:21.787408 19160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:21.787592 19088 raft_consensus.cc:3037] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:21.787645 19160 raft_consensus.cc:3032] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:21.787765 19160 raft_consensus.cc:729] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Running, Role: LEADER
I20240502 14:15:21.788022 19160 consensus_queue.cc:260] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:21.788383 19160 raft_consensus.cc:3037] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:21.790071 19088 raft_consensus.cc:2454] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 2.
I20240502 14:15:21.790427 18982 leader_election.cc:302] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:21.790683 19296 raft_consensus.cc:2781] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:21.790812 19296 raft_consensus.cc:686] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Becoming Leader. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:21.790953 19160 raft_consensus.cc:2454] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 2.
I20240502 14:15:21.791036 19296 consensus_queue.cc:237] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:21.793643 18905 catalog_manager.cc:5484] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: term changed from 1 to 2, leader changed from 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) to 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65). New cstate: current_term: 2 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:21.879199 19424 heartbeater.cc:503] Master 127.15.74.126:42275 was elected leader, sending a full tablet report...
I20240502 14:15:21.914522 19600 ts_tablet_manager.cc:902] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66: Initiating tablet copy from peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141)
I20240502 14:15:21.915808 19600 tablet_copy_client.cc:308] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.65:41141
I20240502 14:15:21.916064 19500 heartbeater.cc:503] Master 127.15.74.126:42275 was elected leader, sending a full tablet report...
I20240502 14:15:21.921712 19026 tablet_copy_service.cc:140] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Received BeginTabletCopySession request for tablet 8f465cbf06f4420c8642e8092ce8e1ed from peer eac995eafe3e4c388a7caf591d6f1c66 ({username='slave'} at 127.0.0.1:54324)
I20240502 14:15:21.921835 19026 tablet_copy_service.cc:161] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Beginning new tablet copy session on tablet 8f465cbf06f4420c8642e8092ce8e1ed from peer eac995eafe3e4c388a7caf591d6f1c66 at {username='slave'} at 127.0.0.1:54324: session id = eac995eafe3e4c388a7caf591d6f1c66-8f465cbf06f4420c8642e8092ce8e1ed
I20240502 14:15:21.924065 19026 tablet_copy_source_session.cc:205] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:21.925132 19600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f465cbf06f4420c8642e8092ce8e1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:21.930272 19600 tablet_copy_client.cc:783] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:21.930501 19600 tablet_copy_client.cc:647] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:21.931792 19604 tablet_copy_client.cc:1002] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Injecting latency into file download: 2000
I20240502 14:15:21.956507 19607 ts_tablet_manager.cc:902] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a: Initiating tablet copy from peer 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:21.957793 19607 tablet_copy_client.cc:308] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.66:44583
I20240502 14:15:21.963234 19101 tablet_copy_service.cc:140] P 80654f573c764bc18e65d583b751997f: Received BeginTabletCopySession request for tablet 99128e8f96dc4e1582ec51cbcb8085e0 from peer a2f6fcadc2e047ddb0c111c95420861a ({username='slave'} at 127.0.0.1:32804)
I20240502 14:15:21.963441 19101 tablet_copy_service.cc:161] P 80654f573c764bc18e65d583b751997f: Beginning new tablet copy session on tablet 99128e8f96dc4e1582ec51cbcb8085e0 from peer a2f6fcadc2e047ddb0c111c95420861a at {username='slave'} at 127.0.0.1:32804: session id = a2f6fcadc2e047ddb0c111c95420861a-99128e8f96dc4e1582ec51cbcb8085e0
I20240502 14:15:21.965849 19101 tablet_copy_source_session.cc:205] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:21.966936 19607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99128e8f96dc4e1582ec51cbcb8085e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:21.971851 19607 tablet_copy_client.cc:783] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:21.972054 19607 tablet_copy_client.cc:647] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:21.973289 19611 tablet_copy_client.cc:1002] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Injecting latency into file download: 2000
I20240502 14:15:21.980988 19574 heartbeater.cc:503] Master 127.15.74.126:42275 was elected leader, sending a full tablet report...
I20240502 14:15:21.995941 19614 ts_tablet_manager.cc:902] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4: Initiating tablet copy from peer 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:21.997646 19614 tablet_copy_client.cc:308] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.67:39767
I20240502 14:15:21.998456 19618 ts_tablet_manager.cc:902] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4: Initiating tablet copy from peer 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:22.003892 19618 tablet_copy_client.cc:308] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.67:39767
I20240502 14:15:22.005867 19175 tablet_copy_service.cc:140] P 4c60ef9c4e8f439facdfdc6fd669ae78: Received BeginTabletCopySession request for tablet 35d01415abc44ec4b52577992892c5b1 from peer d908205271af490dada0a2845191fef4 ({username='slave'} at 127.0.0.1:49510)
I20240502 14:15:22.005985 19175 tablet_copy_service.cc:161] P 4c60ef9c4e8f439facdfdc6fd669ae78: Beginning new tablet copy session on tablet 35d01415abc44ec4b52577992892c5b1 from peer d908205271af490dada0a2845191fef4 at {username='slave'} at 127.0.0.1:49510: session id = d908205271af490dada0a2845191fef4-35d01415abc44ec4b52577992892c5b1
I20240502 14:15:22.006103 19174 tablet_copy_service.cc:140] P 4c60ef9c4e8f439facdfdc6fd669ae78: Received BeginTabletCopySession request for tablet 76a8747fc61249759eb84a0110df4efb from peer d908205271af490dada0a2845191fef4 ({username='slave'} at 127.0.0.1:49510)
I20240502 14:15:22.006233 19174 tablet_copy_service.cc:161] P 4c60ef9c4e8f439facdfdc6fd669ae78: Beginning new tablet copy session on tablet 76a8747fc61249759eb84a0110df4efb from peer d908205271af490dada0a2845191fef4 at {username='slave'} at 127.0.0.1:49510: session id = d908205271af490dada0a2845191fef4-76a8747fc61249759eb84a0110df4efb
I20240502 14:15:22.007856 19175 tablet_copy_source_session.cc:205] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:22.008004 19174 tablet_copy_source_session.cc:205] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:22.008913 19614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35d01415abc44ec4b52577992892c5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:22.009073 19618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a8747fc61249759eb84a0110df4efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:22.013796 19614 tablet_copy_client.cc:783] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:22.013984 19614 tablet_copy_client.cc:647] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:22.014798 19618 tablet_copy_client.cc:783] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:22.014992 19618 tablet_copy_client.cc:647] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:22.015329 19619 tablet_copy_client.cc:1002] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4: tablet copy: Injecting latency into file download: 2000
I20240502 14:15:22.016280 19624 tablet_copy_client.cc:1002] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4: tablet copy: Injecting latency into file download: 2000
I20240502 14:15:22.068019 19628 ts_tablet_manager.cc:902] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66: Initiating tablet copy from peer 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:22.068078 19627 ts_tablet_manager.cc:902] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a: Initiating tablet copy from peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141)
I20240502 14:15:22.071972 19627 tablet_copy_client.cc:308] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.65:41141
I20240502 14:15:22.074136 19628 tablet_copy_client.cc:308] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.66:44583
I20240502 14:15:22.078836 19026 tablet_copy_service.cc:140] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Received BeginTabletCopySession request for tablet 567ee47202bd46048e318b4dbbf3ff03 from peer a2f6fcadc2e047ddb0c111c95420861a ({username='slave'} at 127.0.0.1:54336)
I20240502 14:15:22.078960 19026 tablet_copy_service.cc:161] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Beginning new tablet copy session on tablet 567ee47202bd46048e318b4dbbf3ff03 from peer a2f6fcadc2e047ddb0c111c95420861a at {username='slave'} at 127.0.0.1:54336: session id = a2f6fcadc2e047ddb0c111c95420861a-567ee47202bd46048e318b4dbbf3ff03
I20240502 14:15:22.080315 19101 tablet_copy_service.cc:140] P 80654f573c764bc18e65d583b751997f: Received BeginTabletCopySession request for tablet 73c203a4764245dba0944537c7d10129 from peer eac995eafe3e4c388a7caf591d6f1c66 ({username='slave'} at 127.0.0.1:32816)
I20240502 14:15:22.080443 19101 tablet_copy_service.cc:161] P 80654f573c764bc18e65d583b751997f: Beginning new tablet copy session on tablet 73c203a4764245dba0944537c7d10129 from peer eac995eafe3e4c388a7caf591d6f1c66 at {username='slave'} at 127.0.0.1:32816: session id = eac995eafe3e4c388a7caf591d6f1c66-73c203a4764245dba0944537c7d10129
I20240502 14:15:22.081790 19026 tablet_copy_source_session.cc:205] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:22.082866 19627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567ee47202bd46048e318b4dbbf3ff03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:22.083070 19101 tablet_copy_source_session.cc:205] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:22.084071 19628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c203a4764245dba0944537c7d10129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:22.089032 19627 tablet_copy_client.cc:783] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:22.089229 19627 tablet_copy_client.cc:647] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:22.091080 19637 raft_consensus.cc:979] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: : Instructing follower 80654f573c764bc18e65d583b751997f to start an election
I20240502 14:15:22.091284 19628 tablet_copy_client.cc:783] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:22.091336 19634 tablet_copy_client.cc:1002] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Injecting latency into file download: 2000
I20240502 14:15:22.091346 19637 raft_consensus.cc:1067] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Signalling peer 80654f573c764bc18e65d583b751997f to start an election
I20240502 14:15:22.091485 19628 tablet_copy_client.cc:647] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:22.091874 19088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03"
dest_uuid: "80654f573c764bc18e65d583b751997f"
from {username='slave'} at 127.0.0.1:60998
I20240502 14:15:22.092015 19088 raft_consensus.cc:482] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:22.092100 19088 raft_consensus.cc:3037] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:22.093156 19636 tablet_copy_client.cc:1002] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Injecting latency into file download: 2000
I20240502 14:15:22.094199 19088 raft_consensus.cc:504] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:22.094789 19088 leader_election.cc:288] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 2 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:22.095083 19011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:22.095171 19158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:22.095255 19011 raft_consensus.cc:3032] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:22.095324 19011 raft_consensus.cc:729] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:22.095329 19158 raft_consensus.cc:3037] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:22.095577 19011 consensus_queue.cc:260] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:22.095947 19011 raft_consensus.cc:3037] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:22.097806 19011 raft_consensus.cc:2454] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 2.
I20240502 14:15:22.097913 19158 raft_consensus.cc:2454] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 2.
I20240502 14:15:22.098129 19056 leader_election.cc:302] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:22.098338 19626 raft_consensus.cc:2781] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:22.098515 19626 raft_consensus.cc:686] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 2 LEADER]: Becoming Leader. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:22.098747 19626 consensus_queue.cc:237] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:22.100809 18905 catalog_manager.cc:5484] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f reported cstate change: term changed from 1 to 2, leader changed from 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) to 80654f573c764bc18e65d583b751997f (127.15.74.66). New cstate: current_term: 2 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:22.109203 19088 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03"
dest_uuid: "80654f573c764bc18e65d583b751997f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60996
I20240502 14:15:22.109364 19088 raft_consensus.cc:595] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 2 LEADER]: Received request to transfer leadership
I20240502 14:15:22.115509 19088 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03"
dest_uuid: "80654f573c764bc18e65d583b751997f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60996
I20240502 14:15:22.115679 19088 raft_consensus.cc:595] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 2 LEADER]: Received request to transfer leadership
I20240502 14:15:22.116885 19088 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03"
dest_uuid: "80654f573c764bc18e65d583b751997f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60996
I20240502 14:15:22.117005 19088 raft_consensus.cc:595] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 2 LEADER]: Received request to transfer leadership
W20240502 14:15:22.117405 18962 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 567ee47202bd46048e318b4dbbf3ff03 already in progress
W20240502 14:15:22.117468 18962 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 567ee47202bd46048e318b4dbbf3ff03 already in progress
I20240502 14:15:22.309168 19088 raft_consensus.cc:1261] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 14:15:22.309698 19637 consensus_queue.cc:1035] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:22.311182 19158 raft_consensus.cc:1261] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 14:15:22.311679 19637 consensus_queue.cc:1035] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:22.584074 19160 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:15:22.584638 19626 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:22.587144 19215 raft_consensus.cc:979] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f: : Instructing follower 4c60ef9c4e8f439facdfdc6fd669ae78 to start an election
I20240502 14:15:22.587294 19215 raft_consensus.cc:1067] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 2 LEADER]: Signalling peer 4c60ef9c4e8f439facdfdc6fd669ae78 to start an election
I20240502 14:15:22.589056 19012 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
W20240502 14:15:22.589116 19057 consensus_peers.cc:485] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f -> Peer a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68:41443): Couldn't send request to peer a2f6fcadc2e047ddb0c111c95420861a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:22.589277 19159 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03"
dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
from {username='slave'} at 127.0.0.1:49458
I20240502 14:15:22.589545 19159 raft_consensus.cc:482] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:22.589686 19159 raft_consensus.cc:3037] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:22.589643 19625 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:22.593147 19159 raft_consensus.cc:504] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:22.594233 19159 leader_election.cc:288] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 3 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:22.594323 19011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:22.594633 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:22.594803 19088 raft_consensus.cc:3032] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:22.594923 19088 raft_consensus.cc:729] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:22.595196 19088 consensus_queue.cc:260] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:22.595522 19088 raft_consensus.cc:3037] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:22.595830 19158 raft_consensus.cc:1226] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Rejecting Update request from peer 80654f573c764bc18e65d583b751997f for earlier term 2. Current term is 3. Ops: []
I20240502 14:15:22.597568 19088 raft_consensus.cc:2454] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 3.
I20240502 14:15:22.597863 19131 leader_election.cc:302] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c60ef9c4e8f439facdfdc6fd669ae78, 80654f573c764bc18e65d583b751997f; no voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7
I20240502 14:15:22.598076 19593 raft_consensus.cc:2781] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:22.598245 19593 raft_consensus.cc:686] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 LEADER]: Becoming Leader. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Running, Role: LEADER
I20240502 14:15:22.598492 19593 consensus_queue.cc:237] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:22.601148 18911 catalog_manager.cc:5484] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: term changed from 2 to 3, leader changed from 80654f573c764bc18e65d583b751997f (127.15.74.66) to 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67). New cstate: current_term: 3 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:22.982426 19012 raft_consensus.cc:3037] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:22.985337 19012 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 14:15:22.985875 19648 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:22.993832 19088 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 14:15:22.994362 19596 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240502 14:15:23.003350 19131 consensus_peers.cc:485] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 -> Peer a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68:41443): Couldn't send request to peer a2f6fcadc2e047ddb0c111c95420861a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:23.634343 18967 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:23.637631 19159 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1"
dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
mode: GRACEFUL
new_leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
from {username='slave'} at 127.0.0.1:49498
I20240502 14:15:23.637853 19159 raft_consensus.cc:595] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Received request to transfer leadership to TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7
I20240502 14:15:23.638873 19011 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f465cbf06f4420c8642e8092ce8e1ed"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
mode: GRACEFUL
new_leader_uuid: "80654f573c764bc18e65d583b751997f"
from {username='slave'} at 127.0.0.1:54318
I20240502 14:15:23.639065 19011 raft_consensus.cc:595] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Received request to transfer leadership to TS 80654f573c764bc18e65d583b751997f
I20240502 14:15:23.639518 18967 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 14:15:23.639708 18967 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:23.916021 19654 raft_consensus.cc:979] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: : Instructing follower 80654f573c764bc18e65d583b751997f to start an election
I20240502 14:15:23.916172 19654 raft_consensus.cc:1067] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Signalling peer 80654f573c764bc18e65d583b751997f to start an election
I20240502 14:15:23.916692 19088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8f465cbf06f4420c8642e8092ce8e1ed"
dest_uuid: "80654f573c764bc18e65d583b751997f"
from {username='slave'} at 127.0.0.1:60998
I20240502 14:15:23.916873 19088 raft_consensus.cc:482] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:23.916971 19088 raft_consensus.cc:3037] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:23.919523 19088 raft_consensus.cc:504] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:23.920296 19088 leader_election.cc:288] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 2 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:23.920542 19012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f465cbf06f4420c8642e8092ce8e1ed" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:23.920753 19012 raft_consensus.cc:3032] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:23.920843 19012 raft_consensus.cc:729] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:23.921056 19012 consensus_queue.cc:260] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:23.921483 19012 raft_consensus.cc:3037] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:23.923329 19012 raft_consensus.cc:2454] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 2.
I20240502 14:15:23.923604 19158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f465cbf06f4420c8642e8092ce8e1ed" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:23.923763 19056 leader_election.cc:302] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:23.923782 19158 raft_consensus.cc:3037] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:23.924012 19215 raft_consensus.cc:2781] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:23.927196 19158 raft_consensus.cc:2454] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 2.
I20240502 14:15:23.932181 19600 tablet_copy_client.cc:523] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:23.935340 19600 tablet_bootstrap.cc:492] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66: Bootstrap starting.
I20240502 14:15:23.936349 19215 raft_consensus.cc:686] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 2 LEADER]: Becoming Leader. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:23.936651 19215 consensus_queue.cc:237] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:23.939607 18911 catalog_manager.cc:5484] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f reported cstate change: term changed from 1 to 2, leader changed from 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) to 80654f573c764bc18e65d583b751997f (127.15.74.66). New cstate: current_term: 2 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:23.947402 19088 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f465cbf06f4420c8642e8092ce8e1ed"
dest_uuid: "80654f573c764bc18e65d583b751997f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60996
I20240502 14:15:23.947575 19088 raft_consensus.cc:595] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 2 LEADER]: Received request to transfer leadership
I20240502 14:15:23.953413 19088 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f465cbf06f4420c8642e8092ce8e1ed"
dest_uuid: "80654f573c764bc18e65d583b751997f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60996
I20240502 14:15:23.953545 19088 raft_consensus.cc:595] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 2 LEADER]: Received request to transfer leadership
W20240502 14:15:23.953866 18901 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 8f465cbf06f4420c8642e8092ce8e1ed already in progress [suppressed 1 similar messages]
I20240502 14:15:23.954911 19088 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f465cbf06f4420c8642e8092ce8e1ed"
dest_uuid: "80654f573c764bc18e65d583b751997f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60996
I20240502 14:15:23.955029 19088 raft_consensus.cc:595] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 2 LEADER]: Received request to transfer leadership
W20240502 14:15:23.955384 18962 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 8f465cbf06f4420c8642e8092ce8e1ed already in progress
W20240502 14:15:23.955464 18962 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 8f465cbf06f4420c8642e8092ce8e1ed already in progress
I20240502 14:15:23.969992 19600 tablet_bootstrap.cc:492] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=115 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:23.970395 19600 tablet_bootstrap.cc:492] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66: Bootstrap complete.
I20240502 14:15:23.970670 19600 ts_tablet_manager.cc:1367] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.006s
I20240502 14:15:23.971387 19600 raft_consensus.cc:348] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:23.971592 19600 raft_consensus.cc:729] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eac995eafe3e4c388a7caf591d6f1c66, State: Initialized, Role: LEARNER
I20240502 14:15:23.971799 19600 consensus_queue.cc:260] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:23.972586 19600 ts_tablet_manager.cc:1398] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:23.973193 19026 tablet_copy_service.cc:342] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Request end of tablet copy session eac995eafe3e4c388a7caf591d6f1c66-8f465cbf06f4420c8642e8092ce8e1ed received from {username='slave'} at 127.0.0.1:54324
I20240502 14:15:23.973284 19026 tablet_copy_service.cc:433] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: ending tablet copy session eac995eafe3e4c388a7caf591d6f1c66-8f465cbf06f4420c8642e8092ce8e1ed on tablet 8f465cbf06f4420c8642e8092ce8e1ed with peer eac995eafe3e4c388a7caf591d6f1c66
I20240502 14:15:23.973616 19607 tablet_copy_client.cc:523] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:23.976966 19607 tablet_bootstrap.cc:492] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a: Bootstrap starting.
I20240502 14:15:24.008009 19607 tablet_bootstrap.cc:492] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=93 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:24.008389 19607 tablet_bootstrap.cc:492] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a: Bootstrap complete.
I20240502 14:15:24.008646 19607 ts_tablet_manager.cc:1367] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a: Time spent bootstrapping tablet: real 0.032s user 0.028s sys 0.004s
I20240502 14:15:24.009294 19607 raft_consensus.cc:348] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:24.009544 19607 raft_consensus.cc:729] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2f6fcadc2e047ddb0c111c95420861a, State: Initialized, Role: LEARNER
I20240502 14:15:24.009752 19607 consensus_queue.cc:260] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:24.010480 19607 ts_tablet_manager.cc:1398] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:24.011037 19101 tablet_copy_service.cc:342] P 80654f573c764bc18e65d583b751997f: Request end of tablet copy session a2f6fcadc2e047ddb0c111c95420861a-99128e8f96dc4e1582ec51cbcb8085e0 received from {username='slave'} at 127.0.0.1:32804
I20240502 14:15:24.011137 19101 tablet_copy_service.cc:433] P 80654f573c764bc18e65d583b751997f: ending tablet copy session a2f6fcadc2e047ddb0c111c95420861a-99128e8f96dc4e1582ec51cbcb8085e0 on tablet 99128e8f96dc4e1582ec51cbcb8085e0 with peer a2f6fcadc2e047ddb0c111c95420861a
I20240502 14:15:24.015612 19614 tablet_copy_client.cc:523] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:24.016760 19618 tablet_copy_client.cc:523] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:24.017792 19664 raft_consensus.cc:979] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78: : Instructing follower 1880cd4d2b8e4ec6b42dfd8d2660f5d7 to start an election
I20240502 14:15:24.017875 19664 raft_consensus.cc:1067] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Signalling peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7 to start an election
I20240502 14:15:24.018446 19012 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
from {username='slave'} at 127.0.0.1:54284
I20240502 14:15:24.018596 19012 raft_consensus.cc:482] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:24.018669 19012 raft_consensus.cc:3037] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:24.018766 19614 tablet_bootstrap.cc:492] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4: Bootstrap starting.
I20240502 14:15:24.019619 19618 tablet_bootstrap.cc:492] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4: Bootstrap starting.
I20240502 14:15:24.021242 19012 raft_consensus.cc:504] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:24.022158 19012 leader_election.cc:288] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 3 election: Requested vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:24.022411 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:24.022610 19088 raft_consensus.cc:3037] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:24.025342 19088 raft_consensus.cc:2454] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 3.
I20240502 14:15:24.025751 18982 leader_election.cc:302] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:24.025977 19654 raft_consensus.cc:2781] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:24.026091 19654 raft_consensus.cc:686] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 LEADER]: Becoming Leader. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:24.026331 19654 consensus_queue.cc:237] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:24.027652 19158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:24.027808 19158 raft_consensus.cc:3032] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:24.027873 19158 raft_consensus.cc:729] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Running, Role: LEADER
I20240502 14:15:24.028081 19158 consensus_queue.cc:260] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:24.028573 19158 raft_consensus.cc:3037] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:24.029482 18911 catalog_manager.cc:5484] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: term changed from 2 to 3, leader changed from 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) to 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65). New cstate: current_term: 3 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:24.035123 19158 raft_consensus.cc:2454] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 3.
I20240502 14:15:24.035480 19012 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54276
I20240502 14:15:24.035661 19012 raft_consensus.cc:595] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 LEADER]: Received request to transfer leadership
I20240502 14:15:24.041708 19012 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54276
I20240502 14:15:24.041870 19012 raft_consensus.cc:595] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 LEADER]: Received request to transfer leadership
I20240502 14:15:24.043064 19012 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54276
I20240502 14:15:24.043218 19012 raft_consensus.cc:595] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 LEADER]: Received request to transfer leadership
W20240502 14:15:24.043740 18962 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 35d01415abc44ec4b52577992892c5b1 already in progress
W20240502 14:15:24.043838 18962 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 35d01415abc44ec4b52577992892c5b1 already in progress
I20240502 14:15:24.052393 19614 tablet_bootstrap.cc:492] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=89 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:24.052877 19614 tablet_bootstrap.cc:492] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4: Bootstrap complete.
I20240502 14:15:24.053186 19614 ts_tablet_manager.cc:1367] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4: Time spent bootstrapping tablet: real 0.034s user 0.032s sys 0.005s
I20240502 14:15:24.053931 19614 raft_consensus.cc:348] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:24.054222 19614 raft_consensus.cc:729] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d908205271af490dada0a2845191fef4, State: Initialized, Role: LEARNER
I20240502 14:15:24.054445 19614 consensus_queue.cc:260] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:24.055326 19614 ts_tablet_manager.cc:1398] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:24.056119 19174 tablet_copy_service.cc:342] P 4c60ef9c4e8f439facdfdc6fd669ae78: Request end of tablet copy session d908205271af490dada0a2845191fef4-35d01415abc44ec4b52577992892c5b1 received from {username='slave'} at 127.0.0.1:49510
I20240502 14:15:24.056205 19174 tablet_copy_service.cc:433] P 4c60ef9c4e8f439facdfdc6fd669ae78: ending tablet copy session d908205271af490dada0a2845191fef4-35d01415abc44ec4b52577992892c5b1 on tablet 35d01415abc44ec4b52577992892c5b1 with peer d908205271af490dada0a2845191fef4
I20240502 14:15:24.060611 19398 raft_consensus.cc:1203] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.23->[1.24-1.24] Dedup: 1.24->[]
I20240502 14:15:24.063836 19618 tablet_bootstrap.cc:492] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:24.064157 19618 tablet_bootstrap.cc:492] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4: Bootstrap complete.
I20240502 14:15:24.064402 19618 ts_tablet_manager.cc:1367] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4: Time spent bootstrapping tablet: real 0.045s user 0.035s sys 0.000s
I20240502 14:15:24.064967 19618 raft_consensus.cc:348] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:24.065135 19618 raft_consensus.cc:729] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d908205271af490dada0a2845191fef4, State: Initialized, Role: LEARNER
I20240502 14:15:24.065373 19618 consensus_queue.cc:260] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:24.066293 19618 ts_tablet_manager.cc:1398] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:24.066918 19174 tablet_copy_service.cc:342] P 4c60ef9c4e8f439facdfdc6fd669ae78: Request end of tablet copy session d908205271af490dada0a2845191fef4-76a8747fc61249759eb84a0110df4efb received from {username='slave'} at 127.0.0.1:49510
I20240502 14:15:24.067019 19174 tablet_copy_service.cc:433] P 4c60ef9c4e8f439facdfdc6fd669ae78: ending tablet copy session d908205271af490dada0a2845191fef4-76a8747fc61249759eb84a0110df4efb on tablet 76a8747fc61249759eb84a0110df4efb with peer d908205271af490dada0a2845191fef4
I20240502 14:15:24.091656 19627 tablet_copy_client.cc:523] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:24.093498 19628 tablet_copy_client.cc:523] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:24.095065 19627 tablet_bootstrap.cc:492] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a: Bootstrap starting.
I20240502 14:15:24.096848 19628 tablet_bootstrap.cc:492] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66: Bootstrap starting.
I20240502 14:15:24.135023 19628 tablet_bootstrap.cc:492] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=109 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:24.135497 19627 tablet_bootstrap.cc:492] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=105 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:24.135530 19628 tablet_bootstrap.cc:492] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66: Bootstrap complete.
I20240502 14:15:24.135857 19627 tablet_bootstrap.cc:492] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a: Bootstrap complete.
I20240502 14:15:24.136024 19628 ts_tablet_manager.cc:1367] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66: Time spent bootstrapping tablet: real 0.039s user 0.034s sys 0.005s
I20240502 14:15:24.136090 19627 ts_tablet_manager.cc:1367] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a: Time spent bootstrapping tablet: real 0.041s user 0.039s sys 0.005s
I20240502 14:15:24.136730 19627 raft_consensus.cc:348] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:24.136842 19628 raft_consensus.cc:348] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:24.136983 19627 raft_consensus.cc:729] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2f6fcadc2e047ddb0c111c95420861a, State: Initialized, Role: LEARNER
I20240502 14:15:24.137085 19628 raft_consensus.cc:729] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eac995eafe3e4c388a7caf591d6f1c66, State: Initialized, Role: LEARNER
I20240502 14:15:24.137230 19627 consensus_queue.cc:260] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:24.137252 19628 consensus_queue.cc:260] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:24.137845 19628 ts_tablet_manager.cc:1398] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:24.138150 19627 ts_tablet_manager.cc:1398] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:24.138556 19101 tablet_copy_service.cc:342] P 80654f573c764bc18e65d583b751997f: Request end of tablet copy session eac995eafe3e4c388a7caf591d6f1c66-73c203a4764245dba0944537c7d10129 received from {username='slave'} at 127.0.0.1:32816
I20240502 14:15:24.138648 19101 tablet_copy_service.cc:433] P 80654f573c764bc18e65d583b751997f: ending tablet copy session eac995eafe3e4c388a7caf591d6f1c66-73c203a4764245dba0944537c7d10129 on tablet 73c203a4764245dba0944537c7d10129 with peer eac995eafe3e4c388a7caf591d6f1c66
I20240502 14:15:24.139103 19026 tablet_copy_service.cc:342] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Request end of tablet copy session a2f6fcadc2e047ddb0c111c95420861a-567ee47202bd46048e318b4dbbf3ff03 received from {username='slave'} at 127.0.0.1:54336
I20240502 14:15:24.139189 19026 tablet_copy_service.cc:433] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: ending tablet copy session a2f6fcadc2e047ddb0c111c95420861a-567ee47202bd46048e318b4dbbf3ff03 on tablet 567ee47202bd46048e318b4dbbf3ff03 with peer a2f6fcadc2e047ddb0c111c95420861a
I20240502 14:15:24.369954 19549 raft_consensus.cc:1203] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240502 14:15:24.376622 19474 raft_consensus.cc:1203] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25] Dedup: 1.25->[]
I20240502 14:15:24.408262 19090 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240502 14:15:24.408828 19654 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:24.411284 19655 raft_consensus.cc:979] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: : Instructing follower 80654f573c764bc18e65d583b751997f to start an election
I20240502 14:15:24.411454 19654 raft_consensus.cc:1067] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 LEADER]: Signalling peer 80654f573c764bc18e65d583b751997f to start an election
I20240502 14:15:24.412595 19088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1"
dest_uuid: "80654f573c764bc18e65d583b751997f"
from {username='slave'} at 127.0.0.1:60998
I20240502 14:15:24.413688 19088 raft_consensus.cc:482] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:24.413816 19088 raft_consensus.cc:3037] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:24.416515 19088 raft_consensus.cc:504] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:24.417692 19088 leader_election.cc:288] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 4 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:24.417853 19012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:24.418041 19012 raft_consensus.cc:3032] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:24.418107 19012 raft_consensus.cc:729] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:24.418329 19012 consensus_queue.cc:260] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:24.418710 19012 raft_consensus.cc:3037] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:24.421223 19012 raft_consensus.cc:2454] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 4.
I20240502 14:15:24.417888 19159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:24.421494 19159 raft_consensus.cc:3037] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:24.421648 19056 leader_election.cc:302] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:24.421959 19215 raft_consensus.cc:2781] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:24.422088 19215 raft_consensus.cc:686] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 4 LEADER]: Becoming Leader. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:24.422333 19215 consensus_queue.cc:237] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:24.426148 18905 catalog_manager.cc:5484] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f reported cstate change: term changed from 3 to 4, leader changed from 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) to 80654f573c764bc18e65d583b751997f (127.15.74.66). New cstate: current_term: 4 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:24.431725 19159 raft_consensus.cc:2454] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 4.
I20240502 14:15:24.463130 19158 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:15:24.463662 19215 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:24.465801 19656 raft_consensus.cc:979] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f: : Instructing follower 4c60ef9c4e8f439facdfdc6fd669ae78 to start an election
I20240502 14:15:24.466008 19215 raft_consensus.cc:1067] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 2 LEADER]: Signalling peer 4c60ef9c4e8f439facdfdc6fd669ae78 to start an election
I20240502 14:15:24.466643 19159 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8f465cbf06f4420c8642e8092ce8e1ed"
dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
from {username='slave'} at 127.0.0.1:49458
I20240502 14:15:24.466796 19159 raft_consensus.cc:482] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:24.466876 19159 raft_consensus.cc:3037] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:24.466960 19474 raft_consensus.cc:3037] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66 [term 1 LEARNER]: Advancing to term 2
I20240502 14:15:24.467028 19012 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:15:24.467978 19215 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:24.469317 19159 raft_consensus.cc:504] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:24.469506 19474 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66 [term 2 LEARNER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:15:24.469784 19158 raft_consensus.cc:1226] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Rejecting Update request from peer 80654f573c764bc18e65d583b751997f for earlier term 2. Current term is 3. Ops: []
I20240502 14:15:24.470180 19159 leader_election.cc:288] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 3 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:24.470369 19215 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:24.470468 19011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f465cbf06f4420c8642e8092ce8e1ed" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:24.470744 19090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f465cbf06f4420c8642e8092ce8e1ed" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:24.470901 19090 raft_consensus.cc:3032] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:24.470958 19090 raft_consensus.cc:729] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:24.471160 19090 consensus_queue.cc:260] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:24.471498 19090 raft_consensus.cc:3037] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:24.474946 19090 raft_consensus.cc:2454] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 3.
I20240502 14:15:24.475358 19131 leader_election.cc:302] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c60ef9c4e8f439facdfdc6fd669ae78, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:24.475617 19648 raft_consensus.cc:2781] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:24.475734 19648 raft_consensus.cc:686] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 LEADER]: Becoming Leader. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Running, Role: LEADER
I20240502 14:15:24.475968 19648 consensus_queue.cc:237] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:24.478854 18905 catalog_manager.cc:5484] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: term changed from 2 to 3, leader changed from 80654f573c764bc18e65d583b751997f (127.15.74.66) to 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67). New cstate: current_term: 3 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:15:24.475611 19678 consensus_queue.cc:1159] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" responder_term: 3 status { last_received { term: 2 index: 27 } last_committed_idx: 26 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 80654f573c764bc18e65d583b751997f for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240502 14:15:24.535758 19656 raft_consensus.cc:1050] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f: attempting to promote NON_VOTER a2f6fcadc2e047ddb0c111c95420861a to VOTER
I20240502 14:15:24.536660 19656 consensus_queue.cc:237] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:24.539013 19398 raft_consensus.cc:3037] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [term 1 LEARNER]: Advancing to term 3
I20240502 14:15:24.539021 19012 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 14:15:24.539356 19158 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 14:15:24.539718 19656 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 14:15:24.539920 19397 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 1 LEARNER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 14:15:24.540354 19684 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 14:15:24.540644 19215 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 14:15:24.542009 19398 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [term 3 LEARNER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:15:24.542577 19664 consensus_queue.cc:1037] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:24.552882 19215 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index 24 to 25, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.553933 19664 raft_consensus.cc:1050] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78: attempting to promote NON_VOTER a2f6fcadc2e047ddb0c111c95420861a to VOTER
I20240502 14:15:24.554677 19664 consensus_queue.cc:237] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 27, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:24.557202 19012 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.558161 19088 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 14:15:24.558297 19011 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 14:15:24.558413 19397 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [term 3 LEARNER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 14:15:24.558692 19691 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:24.558838 19398 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.559808 19664 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:24.560163 19664 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:24.561955 19158 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.562269 18905 catalog_manager.cc:5484] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: config changed from index 24 to 25, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.570367 19088 consensus_queue.cc:237] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:24.570941 19648 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 26 to 29, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.571791 19397 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 26 to 29, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.572304 19011 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 14:15:24.572400 19398 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 14:15:24.572881 19678 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:24.573846 19090 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 26 to 29, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.574401 19678 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:24.576982 18906 catalog_manager.cc:5484] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a reported cstate change: config changed from index 26 to 29, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.577723 19012 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 26 to 29, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.579969 19656 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, VOTER 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) evicted. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.581501 19011 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, VOTER 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) evicted. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.583709 18901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99128e8f96dc4e1582ec51cbcb8085e0 with cas_config_opid_index 25: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:24.584625 19398 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, VOTER 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) evicted. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.589123 18911 catalog_manager.cc:5484] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a reported cstate change: config changed from index 25 to 26, VOTER 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) evicted. New cstate: current_term: 1 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.590333 19158 consensus_queue.cc:237] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:24.591836 19011 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:15:24.592270 19691 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:24.593991 19155 tablet_service.cc:1514] Processing DeleteTablet for tablet 99128e8f96dc4e1582ec51cbcb8085e0 with delete_type TABLET_DATA_TOMBSTONED (TS 4c60ef9c4e8f439facdfdc6fd669ae78 not found in new config with opid_index 26) from {username='slave'} at 127.0.0.1:49446
I20240502 14:15:24.594663 19398 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [term 3 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:15:24.595526 19694 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:24.595441 19648 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.597616 19398 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.598416 18898 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 567ee47202bd46048e318b4dbbf3ff03 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:24.598729 19011 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:24.604056 18906 catalog_manager.cc:5484] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: config changed from index 29 to 30, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New cstate: current_term: 3 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:24.606138 19695 tablet_replica.cc:330] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78: stopping tablet replica
I20240502 14:15:24.606882 19695 raft_consensus.cc:2229] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:24.607226 19695 raft_consensus.cc:2258] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:24.608777 19695 ts_tablet_manager.cc:1875] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:24.612255 19080 tablet_service.cc:1514] Processing DeleteTablet for tablet 567ee47202bd46048e318b4dbbf3ff03 with delete_type TABLET_DATA_TOMBSTONED (TS 80654f573c764bc18e65d583b751997f not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:60996
I20240502 14:15:24.613796 19697 tablet_replica.cc:330] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f: stopping tablet replica
I20240502 14:15:24.613988 19697 raft_consensus.cc:2229] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:24.614334 19697 raft_consensus.cc:2258] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:24.615145 19697 ts_tablet_manager.cc:1875] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:24.616919 19695 ts_tablet_manager.cc:1888] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.25
I20240502 14:15:24.617020 19695 log.cc:1196] T 99128e8f96dc4e1582ec51cbcb8085e0 P 4c60ef9c4e8f439facdfdc6fd669ae78: Deleting WAL directory at /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-2-root/wals/99128e8f96dc4e1582ec51cbcb8085e0
I20240502 14:15:24.618117 18898 catalog_manager.cc:4830] TS 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767): tablet 99128e8f96dc4e1582ec51cbcb8085e0 (table test-workload [id=682791e08deb409e86af13a3a7eb1967]) successfully deleted
I20240502 14:15:24.620882 19697 ts_tablet_manager.cc:1888] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.29
I20240502 14:15:24.621027 19697 log.cc:1196] T 567ee47202bd46048e318b4dbbf3ff03 P 80654f573c764bc18e65d583b751997f: Deleting WAL directory at /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-1-root/wals/567ee47202bd46048e318b4dbbf3ff03
I20240502 14:15:24.621683 18901 catalog_manager.cc:4830] TS 80654f573c764bc18e65d583b751997f (127.15.74.66:44583): tablet 567ee47202bd46048e318b4dbbf3ff03 (table test-workload [id=682791e08deb409e86af13a3a7eb1967]) successfully deleted
I20240502 14:15:24.808584 19549 raft_consensus.cc:3037] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4 [term 2 LEARNER]: Advancing to term 4
I20240502 14:15:24.810959 19549 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4 [term 4 LEARNER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240502 14:15:24.811353 19215 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:24.814232 19215 raft_consensus.cc:1050] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f: attempting to promote NON_VOTER d908205271af490dada0a2845191fef4 to VOTER
W20240502 14:15:24.814350 19215 raft_consensus.cc:1054] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f: Unable to promote non-voter d908205271af490dada0a2845191fef4: Illegal state: Leader has not yet committed an operation in its own term
I20240502 14:15:24.859210 19694 raft_consensus.cc:1050] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78: attempting to promote NON_VOTER d908205271af490dada0a2845191fef4 to VOTER
I20240502 14:15:24.860245 19694 consensus_queue.cc:237] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:24.862511 19088 raft_consensus.cc:1261] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 14:15:24.862567 19549 raft_consensus.cc:1261] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [term 1 LEARNER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 14:15:24.863067 19011 raft_consensus.cc:1261] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 14:15:24.863034 19694 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:24.863619 19694 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:24.863931 19696 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:24.872334 19159 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240502 14:15:24.872956 19215 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:24.876267 19696 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.876875 19011 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.881378 18906 catalog_manager.cc:5484] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: config changed from index 26 to 27, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.885584 19088 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.895555 19011 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240502 14:15:24.895556 19548 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.896131 19215 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:24.898857 19215 raft_consensus.cc:1050] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f: attempting to promote NON_VOTER d908205271af490dada0a2845191fef4 to VOTER
I20240502 14:15:24.899709 19215 consensus_queue.cc:237] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 4.28, Last appended by leader: 27, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:24.902324 19012 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240502 14:15:24.902437 19548 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4 [term 4 LEARNER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240502 14:15:24.902871 19215 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:24.902978 19158 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240502 14:15:24.903056 19678 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:24.903506 19159 consensus_queue.cc:237] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:24.903836 19678 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:24.905184 19088 raft_consensus.cc:1261] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 14:15:24.905270 19548 raft_consensus.cc:1261] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [term 1 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 14:15:24.905637 19694 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:24.905977 19696 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:24.907521 19656 raft_consensus.cc:2932] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 4 LEADER]: Committing config change with OpId 4.29: config changed from index 26 to 29, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.908080 19694 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.908643 19549 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.912120 19549 raft_consensus.cc:2932] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4 [term 4 FOLLOWER]: Committing config change with OpId 4.29: config changed from index 26 to 29, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.912498 19088 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.913148 18905 catalog_manager.cc:5484] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 reported cstate change: config changed from index 27 to 28, VOTER 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) evicted. New cstate: current_term: 1 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.917176 19005 tablet_service.cc:1514] Processing DeleteTablet for tablet 76a8747fc61249759eb84a0110df4efb with delete_type TABLET_DATA_TOMBSTONED (TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:54276
I20240502 14:15:24.917825 19708 tablet_replica.cc:330] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: stopping tablet replica
I20240502 14:15:24.918210 19708 raft_consensus.cc:2229] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:24.918511 19708 raft_consensus.cc:2258] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:24.918629 18910 catalog_manager.cc:5484] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4 reported cstate change: config changed from index 26 to 29, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.919523 18898 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 76a8747fc61249759eb84a0110df4efb with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:24.919888 19012 raft_consensus.cc:2932] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Committing config change with OpId 4.29: config changed from index 26 to 29, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.921064 19159 raft_consensus.cc:2932] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Committing config change with OpId 4.29: config changed from index 26 to 29, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.924590 19708 ts_tablet_manager.cc:1875] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:24.929961 19090 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 14:15:24.930547 19694 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:24.930426 19088 consensus_queue.cc:237] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:24.931509 19708 ts_tablet_manager.cc:1888] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240502 14:15:24.931661 19708 log.cc:1196] T 76a8747fc61249759eb84a0110df4efb P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Deleting WAL directory at /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-0-root/wals/76a8747fc61249759eb84a0110df4efb
I20240502 14:15:24.932905 19548 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4 [term 4 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240502 14:15:24.933598 19159 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240502 14:15:24.933542 19656 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:24.934136 19684 raft_consensus.cc:1050] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f: attempting to promote NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 to VOTER
I20240502 14:15:24.935159 19656 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:24.935011 19684 consensus_queue.cc:237] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
I20240502 14:15:24.937255 19656 raft_consensus.cc:2932] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 4 LEADER]: Committing config change with OpId 4.30: config changed from index 29 to 30, VOTER 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.937798 19012 raft_consensus.cc:3037] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:24.939137 19548 raft_consensus.cc:2932] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 29 to 30, VOTER 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.932639 18900 catalog_manager.cc:4830] TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141): tablet 76a8747fc61249759eb84a0110df4efb (table test-workload [id=682791e08deb409e86af13a3a7eb1967]) successfully deleted
I20240502 14:15:24.945253 18901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 35d01415abc44ec4b52577992892c5b1 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:24.947966 19474 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [term 1 LEARNER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 14:15:24.948956 19158 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 14:15:24.948797 19159 raft_consensus.cc:2932] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 29 to 30, VOTER 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:24.950388 19011 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 14:15:24.951653 18906 catalog_manager.cc:5484] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f reported cstate change: config changed from index 29 to 30, VOTER 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) evicted. New cstate: current_term: 4 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:24.952387 19012 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 14:15:24.949512 19684 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.003s
I20240502 14:15:24.952955 19694 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:24.953032 19678 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:24.953850 19656 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:24.956249 19005 tablet_service.cc:1514] Processing DeleteTablet for tablet 35d01415abc44ec4b52577992892c5b1 with delete_type TABLET_DATA_TOMBSTONED (TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:54276
I20240502 14:15:24.956539 19708 tablet_replica.cc:330] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: stopping tablet replica
I20240502 14:15:24.956728 19708 raft_consensus.cc:2229] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:24.956943 19708 raft_consensus.cc:2258] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:24.957788 19708 ts_tablet_manager.cc:1875] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:24.963701 19708 ts_tablet_manager.cc:1888] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.29
I20240502 14:15:24.964584 19708 log.cc:1196] T 35d01415abc44ec4b52577992892c5b1 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Deleting WAL directory at /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-0-root/wals/35d01415abc44ec4b52577992892c5b1
I20240502 14:15:24.965376 18900 catalog_manager.cc:4830] TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141): tablet 35d01415abc44ec4b52577992892c5b1 (table test-workload [id=682791e08deb409e86af13a3a7eb1967]) successfully deleted
I20240502 14:15:24.974942 19684 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:24.976137 19474 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:24.979302 19473 raft_consensus.cc:3037] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66 [term 2 LEARNER]: Advancing to term 3
I20240502 14:15:24.980710 19158 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:24.982407 19473 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66 [term 3 LEARNER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 14:15:24.982913 19694 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:24.984192 18906 catalog_manager.cc:5484] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 reported cstate change: config changed from index 25 to 26, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:24.985179 19011 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:24.985730 19696 raft_consensus.cc:1050] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78: attempting to promote NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 to VOTER
I20240502 14:15:24.986513 19696 consensus_queue.cc:237] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 27, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
I20240502 14:15:24.988629 19012 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 14:15:24.989415 19090 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 14:15:24.989356 19696 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:24.989745 19473 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66 [term 3 LEARNER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 14:15:24.990806 19088 consensus_queue.cc:237] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
I20240502 14:15:24.991468 19696 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:24.992406 19648 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:24.993155 19473 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 14:15:24.994063 19684 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:24.994776 19648 raft_consensus.cc:2932] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 26 to 29, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:24.996573 19012 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 14:15:24.997247 19684 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:24.997808 19656 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:24.999197 19016 raft_consensus.cc:2932] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 26 to 29, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:25.000154 19088 raft_consensus.cc:2932] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 26 to 29, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:25.000625 18905 catalog_manager.cc:5484] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: config changed from index 26 to 29, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:25.002965 19473 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:25.003952 18901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 73c203a4764245dba0944537c7d10129 with cas_config_opid_index 26: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:25.007094 19011 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:25.007231 18906 catalog_manager.cc:5484] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f reported cstate change: config changed from index 26 to 27, VOTER 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) evicted. New cstate: current_term: 1 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:25.007933 19474 raft_consensus.cc:2932] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 26 to 29, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:25.012457 19155 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c203a4764245dba0944537c7d10129 with delete_type TABLET_DATA_TOMBSTONED (TS 4c60ef9c4e8f439facdfdc6fd669ae78 not found in new config with opid_index 27) from {username='slave'} at 127.0.0.1:49446
I20240502 14:15:25.012830 19695 tablet_replica.cc:330] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78: stopping tablet replica
I20240502 14:15:25.012696 19158 consensus_queue.cc:237] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
I20240502 14:15:25.013041 19695 raft_consensus.cc:2229] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:25.013370 19695 raft_consensus.cc:2258] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:25.014343 19695 ts_tablet_manager.cc:1875] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:25.014874 19011 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:15:25.015014 19474 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66 [term 3 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:15:25.015619 19696 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:25.015861 19648 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:25.017994 19694 raft_consensus.cc:2932] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:25.018983 19474 raft_consensus.cc:2932] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:25.019930 19011 raft_consensus.cc:2932] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:25.021539 19695 ts_tablet_manager.cc:1888] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.26
I20240502 14:15:25.021695 19695 log.cc:1196] T 73c203a4764245dba0944537c7d10129 P 4c60ef9c4e8f439facdfdc6fd669ae78: Deleting WAL directory at /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-2-root/wals/73c203a4764245dba0944537c7d10129
I20240502 14:15:25.022337 18898 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f465cbf06f4420c8642e8092ce8e1ed with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:25.022747 18898 catalog_manager.cc:4830] TS 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767): tablet 73c203a4764245dba0944537c7d10129 (table test-workload [id=682791e08deb409e86af13a3a7eb1967]) successfully deleted
I20240502 14:15:25.023408 18906 catalog_manager.cc:5484] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: config changed from index 29 to 30, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New cstate: current_term: 3 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:25.027331 19080 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f465cbf06f4420c8642e8092ce8e1ed with delete_type TABLET_DATA_TOMBSTONED (TS 80654f573c764bc18e65d583b751997f not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:60996
I20240502 14:15:25.027647 19697 tablet_replica.cc:330] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f: stopping tablet replica
I20240502 14:15:25.027843 19697 raft_consensus.cc:2229] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:25.028152 19697 raft_consensus.cc:2258] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:25.028887 19697 ts_tablet_manager.cc:1875] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:25.033133 19697 ts_tablet_manager.cc:1888] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.29
I20240502 14:15:25.033217 19697 log.cc:1196] T 8f465cbf06f4420c8642e8092ce8e1ed P 80654f573c764bc18e65d583b751997f: Deleting WAL directory at /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-1-root/wals/8f465cbf06f4420c8642e8092ce8e1ed
I20240502 14:15:25.033731 18901 catalog_manager.cc:4830] TS 80654f573c764bc18e65d583b751997f (127.15.74.66:44583): tablet 8f465cbf06f4420c8642e8092ce8e1ed (table test-workload [id=682791e08deb409e86af13a3a7eb1967]) successfully deleted
I20240502 14:15:25.640020 18967 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:25.651542 19088 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b"
dest_uuid: "80654f573c764bc18e65d583b751997f"
mode: GRACEFUL
new_leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
from {username='slave'} at 127.0.0.1:34074
I20240502 14:15:25.651715 19088 raft_consensus.cc:595] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Received request to transfer leadership to TS 4c60ef9c4e8f439facdfdc6fd669ae78
I20240502 14:15:25.652639 19158 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03"
dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
mode: GRACEFUL
new_leader_uuid: "a2f6fcadc2e047ddb0c111c95420861a"
from {username='slave'} at 127.0.0.1:49498
I20240502 14:15:25.652796 19158 raft_consensus.cc:595] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 LEADER]: Received request to transfer leadership to TS a2f6fcadc2e047ddb0c111c95420861a
I20240502 14:15:25.653721 19088 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73c203a4764245dba0944537c7d10129"
dest_uuid: "80654f573c764bc18e65d583b751997f"
mode: GRACEFUL
new_leader_uuid: "eac995eafe3e4c388a7caf591d6f1c66"
from {username='slave'} at 127.0.0.1:34074
I20240502 14:15:25.653849 19088 raft_consensus.cc:595] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Received request to transfer leadership to TS eac995eafe3e4c388a7caf591d6f1c66
I20240502 14:15:25.654682 19158 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "76a8747fc61249759eb84a0110df4efb"
dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
mode: GRACEFUL
new_leader_uuid: "d908205271af490dada0a2845191fef4"
from {username='slave'} at 127.0.0.1:49498
I20240502 14:15:25.654806 19158 raft_consensus.cc:595] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Received request to transfer leadership to TS d908205271af490dada0a2845191fef4
I20240502 14:15:25.655618 19088 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f260d59158a344beb686ea0f9746f743"
dest_uuid: "80654f573c764bc18e65d583b751997f"
mode: GRACEFUL
new_leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
from {username='slave'} at 127.0.0.1:34074
I20240502 14:15:25.655757 19088 raft_consensus.cc:595] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Received request to transfer leadership to TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7
I20240502 14:15:25.656105 18967 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240502 14:15:25.656388 18967 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:25.687197 19717 raft_consensus.cc:979] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f: : Instructing follower 1880cd4d2b8e4ec6b42dfd8d2660f5d7 to start an election
I20240502 14:15:25.687345 19717 raft_consensus.cc:1067] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Signalling peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7 to start an election
I20240502 14:15:25.687969 19011 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f260d59158a344beb686ea0f9746f743"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
from {username='slave'} at 127.0.0.1:54300
I20240502 14:15:25.688114 19011 raft_consensus.cc:482] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:25.688194 19011 raft_consensus.cc:3037] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:25.690033 19011 raft_consensus.cc:504] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:25.690577 19011 leader_election.cc:288] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 2 election: Requested vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:25.690953 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f260d59158a344beb686ea0f9746f743" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:25.691054 19158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f260d59158a344beb686ea0f9746f743" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:25.691124 19088 raft_consensus.cc:3032] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:25.691165 19158 raft_consensus.cc:3037] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:25.691177 19088 raft_consensus.cc:729] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:25.691613 19088 consensus_queue.cc:260] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:25.692071 19088 raft_consensus.cc:3037] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:25.693251 19158 raft_consensus.cc:2454] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 2.
I20240502 14:15:25.693656 18979 leader_election.cc:302] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78; no voters:
I20240502 14:15:25.693877 19675 raft_consensus.cc:2781] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:25.694286 19675 raft_consensus.cc:686] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Becoming Leader. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:25.694525 19675 consensus_queue.cc:237] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:25.696852 19088 raft_consensus.cc:2454] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 2.
I20240502 14:15:25.697651 18905 catalog_manager.cc:5484] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: term changed from 1 to 2, leader changed from 80654f573c764bc18e65d583b751997f (127.15.74.66) to 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65). New cstate: current_term: 2 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:25.713030 19723 raft_consensus.cc:979] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78: : Instructing follower a2f6fcadc2e047ddb0c111c95420861a to start an election
I20240502 14:15:25.713200 19664 raft_consensus.cc:1067] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 LEADER]: Signalling peer a2f6fcadc2e047ddb0c111c95420861a to start an election
I20240502 14:15:25.713754 19398 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03"
dest_uuid: "a2f6fcadc2e047ddb0c111c95420861a"
from {username='slave'} at 127.0.0.1:36492
I20240502 14:15:25.713951 19398 raft_consensus.cc:482] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:25.714041 19398 raft_consensus.cc:3037] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:25.716104 19398 raft_consensus.cc:504] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:25.716775 19398 leader_election.cc:288] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [CANDIDATE]: Term 4 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:25.717320 19011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03" candidate_uuid: "a2f6fcadc2e047ddb0c111c95420861a" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:25.717561 19011 raft_consensus.cc:3037] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:25.721601 19011 raft_consensus.cc:2454] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f6fcadc2e047ddb0c111c95420861a in term 4.
I20240502 14:15:25.722283 19363 leader_election.cc:302] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, a2f6fcadc2e047ddb0c111c95420861a; no voters:
I20240502 14:15:25.723307 19726 raft_consensus.cc:2781] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:25.724159 19726 raft_consensus.cc:686] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [term 4 LEADER]: Becoming Leader. State: Replica: a2f6fcadc2e047ddb0c111c95420861a, State: Running, Role: LEADER
I20240502 14:15:25.724742 19726 consensus_queue.cc:237] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:25.728492 18905 catalog_manager.cc:5484] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a reported cstate change: term changed from 3 to 4, leader changed from 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) to a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68). New cstate: current_term: 4 leader_uuid: "a2f6fcadc2e047ddb0c111c95420861a" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:25.730727 19158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567ee47202bd46048e318b4dbbf3ff03" candidate_uuid: "a2f6fcadc2e047ddb0c111c95420861a" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:25.730895 19158 raft_consensus.cc:3032] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:25.730962 19158 raft_consensus.cc:729] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Running, Role: LEADER
I20240502 14:15:25.731174 19158 consensus_queue.cc:260] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:25.731570 19158 raft_consensus.cc:3037] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:25.734310 19158 raft_consensus.cc:2454] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f6fcadc2e047ddb0c111c95420861a in term 4.
I20240502 14:15:25.920516 19684 raft_consensus.cc:979] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f: : Instructing follower 4c60ef9c4e8f439facdfdc6fd669ae78 to start an election
I20240502 14:15:25.920670 19684 raft_consensus.cc:1067] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Signalling peer 4c60ef9c4e8f439facdfdc6fd669ae78 to start an election
I20240502 14:15:25.921201 19158 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b"
dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
from {username='slave'} at 127.0.0.1:49458
I20240502 14:15:25.921360 19158 raft_consensus.cc:482] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:25.921439 19158 raft_consensus.cc:3037] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:25.923287 19158 raft_consensus.cc:504] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:25.923854 19158 leader_election.cc:288] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 2 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:25.924289 19011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:25.924314 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:25.924446 19011 raft_consensus.cc:3037] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:25.924490 19088 raft_consensus.cc:3032] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:25.924563 19088 raft_consensus.cc:729] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:25.924813 19088 consensus_queue.cc:260] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:25.925081 19088 raft_consensus.cc:3037] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:25.926236 19011 raft_consensus.cc:2454] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 2.
I20240502 14:15:25.926555 19130 leader_election.cc:302] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78; no voters:
I20240502 14:15:25.926831 19664 raft_consensus.cc:2781] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:25.926868 19088 raft_consensus.cc:2454] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 2.
I20240502 14:15:25.926949 19664 raft_consensus.cc:686] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Becoming Leader. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Running, Role: LEADER
I20240502 14:15:25.927179 19664 consensus_queue.cc:237] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:25.929389 18905 catalog_manager.cc:5484] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: term changed from 1 to 2, leader changed from 80654f573c764bc18e65d583b751997f (127.15.74.66) to 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67). New cstate: current_term: 2 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:26.005384 19723 raft_consensus.cc:979] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78: : Instructing follower d908205271af490dada0a2845191fef4 to start an election
I20240502 14:15:26.005558 19723 raft_consensus.cc:1067] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Signalling peer d908205271af490dada0a2845191fef4 to start an election
I20240502 14:15:26.006090 19548 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "76a8747fc61249759eb84a0110df4efb"
dest_uuid: "d908205271af490dada0a2845191fef4"
from {username='slave'} at 127.0.0.1:60244
I20240502 14:15:26.006278 19548 raft_consensus.cc:482] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:26.006384 19548 raft_consensus.cc:3037] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.009080 19548 raft_consensus.cc:504] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:26.009989 19548 leader_election.cc:288] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [CANDIDATE]: Term 2 election: Requested vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:26.010474 19158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a8747fc61249759eb84a0110df4efb" candidate_uuid: "d908205271af490dada0a2845191fef4" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:26.010751 19158 raft_consensus.cc:3032] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:26.010861 19158 raft_consensus.cc:729] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Running, Role: LEADER
I20240502 14:15:26.011185 19158 consensus_queue.cc:260] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:26.011555 19158 raft_consensus.cc:3037] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:15:26.012323 18962 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240502 14:15:26.014091 18962 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 1cb757082b084620af257fb553d2dd4b, TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7 -> TS eac995eafe3e4c388a7caf591d6f1c66 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
W20240502 14:15:26.014184 18962 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 1cb757082b084620af257fb553d2dd4b, TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7 -> TS eac995eafe3e4c388a7caf591d6f1c66 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:26.014247 19158 raft_consensus.cc:2454] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d908205271af490dada0a2845191fef4 in term 2.
I20240502 14:15:26.014631 19512 leader_election.cc:302] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c60ef9c4e8f439facdfdc6fd669ae78, d908205271af490dada0a2845191fef4; no voters:
I20240502 14:15:26.015251 19731 raft_consensus.cc:2781] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.015625 19731 raft_consensus.cc:686] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [term 2 LEADER]: Becoming Leader. State: Replica: d908205271af490dada0a2845191fef4, State: Running, Role: LEADER
I20240502 14:15:26.015931 19731 consensus_queue.cc:237] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:26.016145 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a8747fc61249759eb84a0110df4efb" candidate_uuid: "d908205271af490dada0a2845191fef4" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
W20240502 14:15:26.016237 18962 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 154f396806954d2eb24baa4a86ba876d, TS 4c60ef9c4e8f439facdfdc6fd669ae78 -> TS a2f6fcadc2e047ddb0c111c95420861a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
W20240502 14:15:26.016283 18962 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 154f396806954d2eb24baa4a86ba876d, TS 4c60ef9c4e8f439facdfdc6fd669ae78 -> TS a2f6fcadc2e047ddb0c111c95420861a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:26.016300 19088 raft_consensus.cc:3037] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.018414 18905 catalog_manager.cc:5484] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 reported cstate change: term changed from 1 to 2, leader changed from 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) to d908205271af490dada0a2845191fef4 (127.15.74.70). New cstate: current_term: 2 leader_uuid: "d908205271af490dada0a2845191fef4" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:26.018690 19088 raft_consensus.cc:2454] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d908205271af490dada0a2845191fef4 in term 2.
W20240502 14:15:26.019521 18962 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 618d11b0b08e4d49ac15d443acd10e2c, TS 80654f573c764bc18e65d583b751997f -> TS d908205271af490dada0a2845191fef4 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
W20240502 14:15:26.019589 18962 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 618d11b0b08e4d49ac15d443acd10e2c, TS 80654f573c764bc18e65d583b751997f -> TS d908205271af490dada0a2845191fef4 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
W20240502 14:15:26.021451 18962 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6ef2dcb2456949cfa34b5c2e8b0d2e6d, TS 4c60ef9c4e8f439facdfdc6fd669ae78 -> TS eac995eafe3e4c388a7caf591d6f1c66 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
W20240502 14:15:26.021536 18962 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6ef2dcb2456949cfa34b5c2e8b0d2e6d, TS 4c60ef9c4e8f439facdfdc6fd669ae78 -> TS eac995eafe3e4c388a7caf591d6f1c66 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
W20240502 14:15:26.022976 18962 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f260d59158a344beb686ea0f9746f743, TS 80654f573c764bc18e65d583b751997f -> TS d908205271af490dada0a2845191fef4 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
W20240502 14:15:26.023037 18962 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f260d59158a344beb686ea0f9746f743, TS 80654f573c764bc18e65d583b751997f -> TS d908205271af490dada0a2845191fef4 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
W20240502 14:15:26.024333 18962 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 73c203a4764245dba0944537c7d10129, TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7 -> TS a2f6fcadc2e047ddb0c111c95420861a move failed, destination replica disappeared from tablet's Raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
W20240502 14:15:26.024389 18962 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 73c203a4764245dba0944537c7d10129, TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7 -> TS a2f6fcadc2e047ddb0c111c95420861a move failed, destination replica disappeared from tablet's Raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
I20240502 14:15:26.088261 19717 raft_consensus.cc:979] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f: : Instructing follower eac995eafe3e4c388a7caf591d6f1c66 to start an election
I20240502 14:15:26.088418 19717 raft_consensus.cc:1067] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Signalling peer eac995eafe3e4c388a7caf591d6f1c66 to start an election
I20240502 14:15:26.088968 19474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73c203a4764245dba0944537c7d10129"
dest_uuid: "eac995eafe3e4c388a7caf591d6f1c66"
from {username='slave'} at 127.0.0.1:45110
I20240502 14:15:26.089128 19474 raft_consensus.cc:482] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:26.089215 19474 raft_consensus.cc:3037] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.091183 19474 raft_consensus.cc:504] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
I20240502 14:15:26.091889 19474 leader_election.cc:288] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [CANDIDATE]: Term 2 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:26.092240 19011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c203a4764245dba0944537c7d10129" candidate_uuid: "eac995eafe3e4c388a7caf591d6f1c66" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:26.092320 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c203a4764245dba0944537c7d10129" candidate_uuid: "eac995eafe3e4c388a7caf591d6f1c66" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:26.092463 19088 raft_consensus.cc:3032] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:26.092463 19011 raft_consensus.cc:3037] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.092559 19088 raft_consensus.cc:729] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:26.092821 19088 consensus_queue.cc:260] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
I20240502 14:15:26.093205 19088 raft_consensus.cc:3037] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.095237 19011 raft_consensus.cc:2454] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eac995eafe3e4c388a7caf591d6f1c66 in term 2.
I20240502 14:15:26.095588 19439 leader_election.cc:302] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, eac995eafe3e4c388a7caf591d6f1c66; no voters:
I20240502 14:15:26.095885 19088 raft_consensus.cc:2454] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eac995eafe3e4c388a7caf591d6f1c66 in term 2.
I20240502 14:15:26.096441 19734 raft_consensus.cc:2781] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.096745 19734 raft_consensus.cc:686] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [term 2 LEADER]: Becoming Leader. State: Replica: eac995eafe3e4c388a7caf591d6f1c66, State: Running, Role: LEADER
I20240502 14:15:26.096964 19734 consensus_queue.cc:237] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
I20240502 14:15:26.098856 18905 catalog_manager.cc:5484] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 reported cstate change: term changed from 1 to 2, leader changed from 80654f573c764bc18e65d583b751997f (127.15.74.66) to eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69). New cstate: current_term: 2 leader_uuid: "eac995eafe3e4c388a7caf591d6f1c66" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:26.163093 19011 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 14:15:26.163093 19088 raft_consensus.cc:1261] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 14:15:26.163754 19675 consensus_queue.cc:1035] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:26.163754 19726 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:15:26.169042 19158 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 14:15:26.169610 19742 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:15:26.172778 19159 raft_consensus.cc:1261] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 14:15:26.173970 19744 consensus_queue.cc:1035] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:26.332401 19011 raft_consensus.cc:1261] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 14:15:26.332953 19723 consensus_queue.cc:1035] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:26.336671 19088 raft_consensus.cc:1261] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 14:15:26.337158 19723 consensus_queue.cc:1035] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:26.443989 19158 raft_consensus.cc:1261] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Refusing update from remote peer d908205271af490dada0a2845191fef4: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 14:15:26.444517 19731 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:26.448731 19088 raft_consensus.cc:1261] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer d908205271af490dada0a2845191fef4: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 14:15:26.449167 19731 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:26.523893 19088 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer eac995eafe3e4c388a7caf591d6f1c66: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 14:15:26.524468 19734 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:26.528131 19011 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Refusing update from remote peer eac995eafe3e4c388a7caf591d6f1c66: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 14:15:26.528630 19756 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:27.034852 19398 consensus_queue.cc:237] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:27.036820 19011 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240502 14:15:27.037369 19742 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:15:27.037510 19158 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240502 14:15:27.038039 19742 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:15:27.039557 19726 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 30 to 32, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
I20240502 14:15:27.040391 19011 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 30 to 32, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
W20240502 14:15:27.042840 19364 consensus_peers.cc:485] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a -> Peer eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69:44877): Couldn't send request to peer eac995eafe3e4c388a7caf591d6f1c66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 567ee47202bd46048e318b4dbbf3ff03. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:27.043365 19159 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 30 to 32, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
I20240502 14:15:27.044600 18906 catalog_manager.cc:5484] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a reported cstate change: config changed from index 30 to 32, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New cstate: current_term: 4 leader_uuid: "a2f6fcadc2e047ddb0c111c95420861a" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:27.044992 19012 consensus_queue.cc:237] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:27.049005 19159 raft_consensus.cc:1261] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:15:27.049685 19744 consensus_queue.cc:1035] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:27.052094 19758 raft_consensus.cc:2932] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:27.052665 19160 raft_consensus.cc:2932] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:27.056895 19088 raft_consensus.cc:1261] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:15:27.057447 19759 consensus_queue.cc:1035] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:27.058523 19011 consensus_queue.cc:237] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:27.061323 19088 raft_consensus.cc:2932] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
W20240502 14:15:27.062660 18982 consensus_peers.cc:485] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 -> Peer a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68:41443): Couldn't send request to peer a2f6fcadc2e047ddb0c111c95420861a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 154f396806954d2eb24baa4a86ba876d. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.064956 18982 consensus_peers.cc:485] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 -> Peer d908205271af490dada0a2845191fef4 (127.15.74.70:35089): Couldn't send request to peer d908205271af490dada0a2845191fef4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f260d59158a344beb686ea0f9746f743. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:27.065883 19159 raft_consensus.cc:1261] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:15:27.065883 19090 raft_consensus.cc:1261] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:15:27.066378 19744 consensus_queue.cc:1035] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:27.066335 18910 catalog_manager.cc:5484] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f reported cstate change: config changed from index -1 to 27, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New cstate: current_term: 2 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:27.066840 19744 consensus_queue.cc:1035] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:27.069564 19759 raft_consensus.cc:2932] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:27.070065 19159 raft_consensus.cc:2932] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:27.075390 19011 consensus_queue.cc:237] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:27.076042 19088 raft_consensus.cc:2932] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:27.077606 19090 raft_consensus.cc:1261] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
W20240502 14:15:27.078002 18982 consensus_peers.cc:485] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 -> Peer eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69:44877): Couldn't send request to peer eac995eafe3e4c388a7caf591d6f1c66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ef2dcb2456949cfa34b5c2e8b0d2e6d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:27.078267 19759 consensus_queue.cc:1035] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240502 14:15:27.078330 19159 raft_consensus.cc:1261] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20240502 14:15:27.078783 19759 consensus_queue.cc:1035] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240502 14:15:27.081727 18911 catalog_manager.cc:5484] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: config changed from index -1 to 27, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New cstate: current_term: 2 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:27.088661 19744 raft_consensus.cc:2932] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index -1 to 24, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
I20240502 14:15:27.089303 19159 raft_consensus.cc:2932] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index -1 to 24, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
I20240502 14:15:27.090121 19090 raft_consensus.cc:2932] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index -1 to 24, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
I20240502 14:15:27.093911 18910 catalog_manager.cc:5484] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: config changed from index -1 to 24, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New cstate: current_term: 1 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:27.099741 19548 consensus_queue.cc:237] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:27.101728 19159 raft_consensus.cc:1261] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Refusing update from remote peer d908205271af490dada0a2845191fef4: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:15:27.102432 19090 raft_consensus.cc:1261] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer d908205271af490dada0a2845191fef4: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:15:27.102346 19773 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:27.103000 19773 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:27.104655 19773 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:27.105392 19159 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:27.106254 19090 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
W20240502 14:15:27.109361 19515 consensus_peers.cc:485] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 -> Peer a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68:41443): Couldn't send request to peer a2f6fcadc2e047ddb0c111c95420861a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76a8747fc61249759eb84a0110df4efb. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:27.111459 18906 catalog_manager.cc:5484] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 reported cstate change: config changed from index 28 to 30, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New cstate: current_term: 2 leader_uuid: "d908205271af490dada0a2845191fef4" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:27.117424 19474 consensus_queue.cc:237] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:27.119813 19011 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Refusing update from remote peer eac995eafe3e4c388a7caf591d6f1c66: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 14:15:27.119979 19090 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer eac995eafe3e4c388a7caf591d6f1c66: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 14:15:27.120486 19756 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:27.120754 19735 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:27.122872 19735 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:27.124238 19011 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
W20240502 14:15:27.126857 19440 consensus_peers.cc:485] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 -> Peer d908205271af490dada0a2845191fef4 (127.15.74.70:35089): Couldn't send request to peer d908205271af490dada0a2845191fef4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73c203a4764245dba0944537c7d10129. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:27.127187 19090 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:27.128284 18911 catalog_manager.cc:5484] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 reported cstate change: config changed from index 27 to 29, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New cstate: current_term: 2 leader_uuid: "eac995eafe3e4c388a7caf591d6f1c66" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:27.570564 19779 ts_tablet_manager.cc:902] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66: Initiating tablet copy from peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141)
I20240502 14:15:27.572292 19779 tablet_copy_client.cc:308] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.65:41141
I20240502 14:15:27.572885 19026 tablet_copy_service.cc:140] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Received BeginTabletCopySession request for tablet 6ef2dcb2456949cfa34b5c2e8b0d2e6d from peer eac995eafe3e4c388a7caf591d6f1c66 ({username='slave'} at 127.0.0.1:54324)
I20240502 14:15:27.573041 19026 tablet_copy_service.cc:161] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Beginning new tablet copy session on tablet 6ef2dcb2456949cfa34b5c2e8b0d2e6d from peer eac995eafe3e4c388a7caf591d6f1c66 at {username='slave'} at 127.0.0.1:54324: session id = eac995eafe3e4c388a7caf591d6f1c66-6ef2dcb2456949cfa34b5c2e8b0d2e6d
I20240502 14:15:27.576018 19026 tablet_copy_source_session.cc:205] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:27.577232 19779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef2dcb2456949cfa34b5c2e8b0d2e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:27.582600 19786 ts_tablet_manager.cc:902] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a: Initiating tablet copy from peer d908205271af490dada0a2845191fef4 (127.15.74.70:35089)
I20240502 14:15:27.587528 19779 tablet_copy_client.cc:783] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:27.587888 19779 tablet_copy_client.cc:647] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:27.589957 19783 tablet_copy_client.cc:1002] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Injecting latency into file download: 2000
I20240502 14:15:27.604683 19787 ts_tablet_manager.cc:902] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66: Initiating tablet copy from peer a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68:41443)
I20240502 14:15:27.608266 19785 ts_tablet_manager.cc:902] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a: Initiating tablet copy from peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141)
I20240502 14:15:27.611789 19786 tablet_copy_client.cc:308] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.70:35089
I20240502 14:15:27.629581 19787 tablet_copy_client.cc:308] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.68:41443
I20240502 14:15:27.639515 19785 tablet_copy_client.cc:308] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.65:41141
I20240502 14:15:27.641618 19559 tablet_copy_service.cc:140] P d908205271af490dada0a2845191fef4: Received BeginTabletCopySession request for tablet 76a8747fc61249759eb84a0110df4efb from peer a2f6fcadc2e047ddb0c111c95420861a ({username='slave'} at 127.0.0.1:49214)
I20240502 14:15:27.641789 19559 tablet_copy_service.cc:161] P d908205271af490dada0a2845191fef4: Beginning new tablet copy session on tablet 76a8747fc61249759eb84a0110df4efb from peer a2f6fcadc2e047ddb0c111c95420861a at {username='slave'} at 127.0.0.1:49214: session id = a2f6fcadc2e047ddb0c111c95420861a-76a8747fc61249759eb84a0110df4efb
I20240502 14:15:27.644093 19026 tablet_copy_service.cc:140] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Received BeginTabletCopySession request for tablet 154f396806954d2eb24baa4a86ba876d from peer a2f6fcadc2e047ddb0c111c95420861a ({username='slave'} at 127.0.0.1:54336)
I20240502 14:15:27.644213 19026 tablet_copy_service.cc:161] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Beginning new tablet copy session on tablet 154f396806954d2eb24baa4a86ba876d from peer a2f6fcadc2e047ddb0c111c95420861a at {username='slave'} at 127.0.0.1:54336: session id = a2f6fcadc2e047ddb0c111c95420861a-154f396806954d2eb24baa4a86ba876d
I20240502 14:15:27.645220 19559 tablet_copy_source_session.cc:205] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:27.646590 19026 tablet_copy_source_session.cc:205] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:27.647716 19785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 154f396806954d2eb24baa4a86ba876d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:27.648299 19786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a8747fc61249759eb84a0110df4efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:27.652799 19409 tablet_copy_service.cc:140] P a2f6fcadc2e047ddb0c111c95420861a: Received BeginTabletCopySession request for tablet 567ee47202bd46048e318b4dbbf3ff03 from peer eac995eafe3e4c388a7caf591d6f1c66 ({username='slave'} at 127.0.0.1:36514)
I20240502 14:15:27.652925 19409 tablet_copy_service.cc:161] P a2f6fcadc2e047ddb0c111c95420861a: Beginning new tablet copy session on tablet 567ee47202bd46048e318b4dbbf3ff03 from peer eac995eafe3e4c388a7caf591d6f1c66 at {username='slave'} at 127.0.0.1:36514: session id = eac995eafe3e4c388a7caf591d6f1c66-567ee47202bd46048e318b4dbbf3ff03
I20240502 14:15:27.653749 19786 tablet_copy_client.cc:783] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:27.653939 19786 tablet_copy_client.cc:647] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:27.655217 19796 tablet_copy_client.cc:1002] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Injecting latency into file download: 2000
I20240502 14:15:27.655718 19409 tablet_copy_source_session.cc:205] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:27.656718 18967 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:27.656828 19787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567ee47202bd46048e318b4dbbf3ff03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:27.657392 19785 tablet_copy_client.cc:783] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:27.657596 19785 tablet_copy_client.cc:647] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:27.659027 19801 tablet_copy_client.cc:1002] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Injecting latency into file download: 2000
I20240502 14:15:27.661067 19012 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
mode: GRACEFUL
new_leader_uuid: "80654f573c764bc18e65d583b751997f"
from {username='slave'} at 127.0.0.1:54318
I20240502 14:15:27.661242 19012 raft_consensus.cc:595] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Received request to transfer leadership to TS 80654f573c764bc18e65d583b751997f
I20240502 14:15:27.661660 19787 tablet_copy_client.cc:783] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:27.661916 19787 tablet_copy_client.cc:647] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:27.662468 19159 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b"
dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
mode: GRACEFUL
new_leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
from {username='slave'} at 127.0.0.1:49498
I20240502 14:15:27.662621 19159 raft_consensus.cc:595] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Received request to transfer leadership to TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7
I20240502 14:15:27.663697 19800 tablet_copy_client.cc:1002] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Injecting latency into file download: 2000
I20240502 14:15:27.664060 19012 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "618d11b0b08e4d49ac15d443acd10e2c"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
mode: GRACEFUL
new_leader_uuid: "80654f573c764bc18e65d583b751997f"
from {username='slave'} at 127.0.0.1:54318
I20240502 14:15:27.664213 19012 raft_consensus.cc:595] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Received request to transfer leadership to TS 80654f573c764bc18e65d583b751997f
I20240502 14:15:27.664680 18967 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 14:15:27.665078 18967 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:27.667546 19804 ts_tablet_manager.cc:902] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4: Initiating tablet copy from peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141)
I20240502 14:15:27.668968 19804 tablet_copy_client.cc:308] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.65:41141
I20240502 14:15:27.671531 19744 raft_consensus.cc:979] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: : Instructing follower 80654f573c764bc18e65d583b751997f to start an election
I20240502 14:15:27.671715 19744 raft_consensus.cc:1067] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Signalling peer 80654f573c764bc18e65d583b751997f to start an election
I20240502 14:15:27.672355 19090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d"
dest_uuid: "80654f573c764bc18e65d583b751997f"
from {username='slave'} at 127.0.0.1:60998
I20240502 14:15:27.672564 19090 raft_consensus.cc:482] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:27.672664 19090 raft_consensus.cc:3037] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:27.674741 19090 raft_consensus.cc:504] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:27.675595 19011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:27.675665 19159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:27.675766 19011 raft_consensus.cc:3032] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:27.675822 19011 raft_consensus.cc:729] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:27.675823 19159 raft_consensus.cc:3037] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:27.676048 19011 consensus_queue.cc:260] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:27.676471 19011 raft_consensus.cc:3037] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:27.676717 19090 leader_election.cc:288] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 3 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:27.678187 19159 raft_consensus.cc:2454] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 3.
I20240502 14:15:27.678654 19011 raft_consensus.cc:2454] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 3.
I20240502 14:15:27.678632 19054 leader_election.cc:302] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c60ef9c4e8f439facdfdc6fd669ae78, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:27.678951 19717 raft_consensus.cc:2781] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:27.679072 19717 raft_consensus.cc:686] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 3 LEADER]: Becoming Leader. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:27.679513 19026 tablet_copy_service.cc:140] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Received BeginTabletCopySession request for tablet f260d59158a344beb686ea0f9746f743 from peer d908205271af490dada0a2845191fef4 ({username='slave'} at 127.0.0.1:54696)
I20240502 14:15:27.679651 19026 tablet_copy_service.cc:161] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Beginning new tablet copy session on tablet f260d59158a344beb686ea0f9746f743 from peer d908205271af490dada0a2845191fef4 at {username='slave'} at 127.0.0.1:54696: session id = d908205271af490dada0a2845191fef4-f260d59158a344beb686ea0f9746f743
I20240502 14:15:27.681954 19026 tablet_copy_source_session.cc:205] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:27.679334 19717 consensus_queue.cc:237] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:27.683152 19804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f260d59158a344beb686ea0f9746f743. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:27.685272 18906 catalog_manager.cc:5484] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f reported cstate change: term changed from 2 to 3, leader changed from 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) to 80654f573c764bc18e65d583b751997f (127.15.74.66). New cstate: current_term: 3 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:27.688446 19804 tablet_copy_client.cc:783] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:27.688717 19804 tablet_copy_client.cc:647] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:27.690251 19808 tablet_copy_client.cc:1002] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4: tablet copy: Injecting latency into file download: 2000
I20240502 14:15:27.702193 19814 ts_tablet_manager.cc:902] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4: Initiating tablet copy from peer eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69:44877)
I20240502 14:15:27.704571 19814 tablet_copy_client.cc:308] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.69:44877
I20240502 14:15:27.710095 19485 tablet_copy_service.cc:140] P eac995eafe3e4c388a7caf591d6f1c66: Received BeginTabletCopySession request for tablet 73c203a4764245dba0944537c7d10129 from peer d908205271af490dada0a2845191fef4 ({username='slave'} at 127.0.0.1:55232)
I20240502 14:15:27.710243 19485 tablet_copy_service.cc:161] P eac995eafe3e4c388a7caf591d6f1c66: Beginning new tablet copy session on tablet 73c203a4764245dba0944537c7d10129 from peer d908205271af490dada0a2845191fef4 at {username='slave'} at 127.0.0.1:55232: session id = d908205271af490dada0a2845191fef4-73c203a4764245dba0944537c7d10129
I20240502 14:15:27.712843 19485 tablet_copy_source_session.cc:205] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:27.713960 19814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c203a4764245dba0944537c7d10129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:27.719326 19814 tablet_copy_client.cc:783] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:27.719547 19814 tablet_copy_client.cc:647] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:27.721386 19818 tablet_copy_client.cc:1002] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4: tablet copy: Injecting latency into file download: 2000
I20240502 14:15:27.740500 19744 raft_consensus.cc:979] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: : Instructing follower 80654f573c764bc18e65d583b751997f to start an election
I20240502 14:15:27.740669 19744 raft_consensus.cc:1067] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Signalling peer 80654f573c764bc18e65d583b751997f to start an election
I20240502 14:15:27.741173 19090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "618d11b0b08e4d49ac15d443acd10e2c"
dest_uuid: "80654f573c764bc18e65d583b751997f"
from {username='slave'} at 127.0.0.1:60998
I20240502 14:15:27.741366 19090 raft_consensus.cc:482] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:27.741446 19090 raft_consensus.cc:3037] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:27.744097 19090 raft_consensus.cc:504] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:27.744765 19090 leader_election.cc:288] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 2 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:27.745227 19159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618d11b0b08e4d49ac15d443acd10e2c" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:27.745273 19012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618d11b0b08e4d49ac15d443acd10e2c" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:27.745406 19159 raft_consensus.cc:3037] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:27.745474 19012 raft_consensus.cc:3032] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:27.745549 19012 raft_consensus.cc:729] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:27.745790 19012 consensus_queue.cc:260] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:27.746196 19012 raft_consensus.cc:3037] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:27.748216 19159 raft_consensus.cc:2454] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 2.
I20240502 14:15:27.748615 19054 leader_election.cc:302] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c60ef9c4e8f439facdfdc6fd669ae78, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:27.748683 19012 raft_consensus.cc:2454] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 2.
I20240502 14:15:27.748901 19717 raft_consensus.cc:2781] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:27.749022 19717 raft_consensus.cc:686] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [term 2 LEADER]: Becoming Leader. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:27.749238 19717 consensus_queue.cc:237] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:27.751560 18906 catalog_manager.cc:5484] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f reported cstate change: term changed from 1 to 2, leader changed from 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) to 80654f573c764bc18e65d583b751997f (127.15.74.66). New cstate: current_term: 2 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:27.976090 19749 raft_consensus.cc:979] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78: : Instructing follower 1880cd4d2b8e4ec6b42dfd8d2660f5d7 to start an election
I20240502 14:15:27.976261 19749 raft_consensus.cc:1067] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Signalling peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7 to start an election
I20240502 14:15:27.976819 19012 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
from {username='slave'} at 127.0.0.1:54284
I20240502 14:15:27.977042 19012 raft_consensus.cc:482] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:27.977138 19012 raft_consensus.cc:3037] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:27.979888 19012 raft_consensus.cc:504] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:27.980787 19159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:27.980791 19090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:27.980964 19159 raft_consensus.cc:3032] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:27.980988 19090 raft_consensus.cc:3037] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:27.981029 19159 raft_consensus.cc:729] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Running, Role: LEADER
I20240502 14:15:27.981365 19012 leader_election.cc:288] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 3 election: Requested vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:27.981264 19159 consensus_queue.cc:260] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:27.981606 19159 raft_consensus.cc:3037] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:27.983508 19090 raft_consensus.cc:2454] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 3.
I20240502 14:15:27.983670 19159 raft_consensus.cc:2454] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 3.
I20240502 14:15:27.983996 18979 leader_election.cc:302] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 4c60ef9c4e8f439facdfdc6fd669ae78; no voters:
I20240502 14:15:27.984261 19744 raft_consensus.cc:2781] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:27.984413 19744 raft_consensus.cc:686] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 LEADER]: Becoming Leader. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:27.984668 19744 consensus_queue.cc:237] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:27.988173 18906 catalog_manager.cc:5484] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: term changed from 2 to 3, leader changed from 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) to 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65). New cstate: current_term: 3 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:28.085130 19011 raft_consensus.cc:1261] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 14:15:28.086433 19717 consensus_queue.cc:1035] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240502 14:15:28.094906 19057 consensus_peers.cc:485] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f -> Peer a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68:41443): Couldn't send request to peer a2f6fcadc2e047ddb0c111c95420861a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:28.095510 19160 raft_consensus.cc:1261] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 14:15:28.096153 19717 consensus_queue.cc:1035] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:28.311383 19012 raft_consensus.cc:1261] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 14:15:28.311914 19717 consensus_queue.cc:1035] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:28.315291 19160 raft_consensus.cc:1261] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 14:15:28.315760 19784 consensus_queue.cc:1035] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:28.386734 19088 raft_consensus.cc:1261] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240502 14:15:28.387267 19823 consensus_queue.cc:1035] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:28.391091 19160 raft_consensus.cc:1261] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240502 14:15:28.391983 19744 consensus_queue.cc:1035] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:29.590714 19779 tablet_copy_client.cc:523] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:29.594218 19779 tablet_bootstrap.cc:492] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66: Bootstrap starting.
I20240502 14:15:29.625645 19779 tablet_bootstrap.cc:492] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=83 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:29.626085 19779 tablet_bootstrap.cc:492] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66: Bootstrap complete.
I20240502 14:15:29.626365 19779 ts_tablet_manager.cc:1367] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.006s
I20240502 14:15:29.627118 19779 raft_consensus.cc:348] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:29.627385 19779 raft_consensus.cc:729] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eac995eafe3e4c388a7caf591d6f1c66, State: Initialized, Role: LEARNER
I20240502 14:15:29.627596 19779 consensus_queue.cc:260] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:29.628206 19779 ts_tablet_manager.cc:1398] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:15:29.628851 19026 tablet_copy_service.cc:342] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Request end of tablet copy session eac995eafe3e4c388a7caf591d6f1c66-6ef2dcb2456949cfa34b5c2e8b0d2e6d received from {username='slave'} at 127.0.0.1:54324
I20240502 14:15:29.628929 19026 tablet_copy_service.cc:433] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: ending tablet copy session eac995eafe3e4c388a7caf591d6f1c66-6ef2dcb2456949cfa34b5c2e8b0d2e6d on tablet 6ef2dcb2456949cfa34b5c2e8b0d2e6d with peer eac995eafe3e4c388a7caf591d6f1c66
I20240502 14:15:29.655737 19786 tablet_copy_client.cc:523] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:29.658823 19786 tablet_bootstrap.cc:492] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a: Bootstrap starting.
I20240502 14:15:29.659919 19785 tablet_copy_client.cc:523] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:29.663264 19785 tablet_bootstrap.cc:492] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a: Bootstrap starting.
I20240502 14:15:29.664515 19787 tablet_copy_client.cc:523] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:29.665611 18967 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:29.667955 19787 tablet_bootstrap.cc:492] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66: Bootstrap starting.
I20240502 14:15:29.669778 19088 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1"
dest_uuid: "80654f573c764bc18e65d583b751997f"
mode: GRACEFUL
new_leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
from {username='slave'} at 127.0.0.1:34074
I20240502 14:15:29.670012 19088 raft_consensus.cc:595] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 4 LEADER]: Received request to transfer leadership to TS 4c60ef9c4e8f439facdfdc6fd669ae78
I20240502 14:15:29.671195 19012 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ef2dcb2456949cfa34b5c2e8b0d2e6d"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
mode: GRACEFUL
new_leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
from {username='slave'} at 127.0.0.1:54318
I20240502 14:15:29.671344 19012 raft_consensus.cc:595] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Received request to transfer leadership to TS 4c60ef9c4e8f439facdfdc6fd669ae78
I20240502 14:15:29.672515 19088 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99128e8f96dc4e1582ec51cbcb8085e0"
dest_uuid: "80654f573c764bc18e65d583b751997f"
mode: GRACEFUL
new_leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
from {username='slave'} at 127.0.0.1:34074
I20240502 14:15:29.672720 19088 raft_consensus.cc:595] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Received request to transfer leadership to TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7
I20240502 14:15:29.673257 18967 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 14:15:29.673596 18967 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:29.690768 19804 tablet_copy_client.cc:523] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:29.694339 19804 tablet_bootstrap.cc:492] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4: Bootstrap starting.
I20240502 14:15:29.708835 19785 tablet_bootstrap.cc:492] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:29.709286 19785 tablet_bootstrap.cc:492] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a: Bootstrap complete.
I20240502 14:15:29.709654 19785 ts_tablet_manager.cc:1367] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a: Time spent bootstrapping tablet: real 0.046s user 0.035s sys 0.009s
I20240502 14:15:29.710408 19785 raft_consensus.cc:348] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:29.710765 19785 raft_consensus.cc:729] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a2f6fcadc2e047ddb0c111c95420861a, State: Initialized, Role: LEARNER
I20240502 14:15:29.710985 19785 consensus_queue.cc:260] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:29.712177 19785 ts_tablet_manager.cc:1398] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:29.712754 19026 tablet_copy_service.cc:342] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Request end of tablet copy session a2f6fcadc2e047ddb0c111c95420861a-154f396806954d2eb24baa4a86ba876d received from {username='slave'} at 127.0.0.1:54336
I20240502 14:15:29.712836 19026 tablet_copy_service.cc:433] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: ending tablet copy session a2f6fcadc2e047ddb0c111c95420861a-154f396806954d2eb24baa4a86ba876d on tablet 154f396806954d2eb24baa4a86ba876d with peer a2f6fcadc2e047ddb0c111c95420861a
I20240502 14:15:29.721885 19814 tablet_copy_client.cc:523] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:29.725296 19814 tablet_bootstrap.cc:492] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4: Bootstrap starting.
I20240502 14:15:29.727312 19786 tablet_bootstrap.cc:492] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:29.727685 19786 tablet_bootstrap.cc:492] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a: Bootstrap complete.
I20240502 14:15:29.727953 19786 ts_tablet_manager.cc:1367] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a: Time spent bootstrapping tablet: real 0.069s user 0.042s sys 0.009s
I20240502 14:15:29.728662 19846 raft_consensus.cc:979] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f: : Instructing follower 1880cd4d2b8e4ec6b42dfd8d2660f5d7 to start an election
I20240502 14:15:29.728806 19828 raft_consensus.cc:1067] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Signalling peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7 to start an election
I20240502 14:15:29.728816 19786 raft_consensus.cc:348] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:29.729132 19786 raft_consensus.cc:729] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a2f6fcadc2e047ddb0c111c95420861a, State: Initialized, Role: LEARNER
I20240502 14:15:29.729498 19011 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99128e8f96dc4e1582ec51cbcb8085e0"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
from {username='slave'} at 127.0.0.1:54300
I20240502 14:15:29.729692 19011 raft_consensus.cc:482] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:29.729780 19011 raft_consensus.cc:3037] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:29.729787 19786 consensus_queue.cc:260] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:29.730492 19786 ts_tablet_manager.cc:1398] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:29.731117 19559 tablet_copy_service.cc:342] P d908205271af490dada0a2845191fef4: Request end of tablet copy session a2f6fcadc2e047ddb0c111c95420861a-76a8747fc61249759eb84a0110df4efb received from {username='slave'} at 127.0.0.1:49214
I20240502 14:15:29.731210 19559 tablet_copy_service.cc:433] P d908205271af490dada0a2845191fef4: ending tablet copy session a2f6fcadc2e047ddb0c111c95420861a-76a8747fc61249759eb84a0110df4efb on tablet 76a8747fc61249759eb84a0110df4efb with peer a2f6fcadc2e047ddb0c111c95420861a
I20240502 14:15:29.732422 19011 raft_consensus.cc:504] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:29.733043 19011 leader_election.cc:288] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 2 election: Requested vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68:41443)
I20240502 14:15:29.737123 19787 tablet_bootstrap.cc:492] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=105 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:29.737505 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99128e8f96dc4e1582ec51cbcb8085e0" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:29.737664 19088 raft_consensus.cc:3032] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:29.737741 19088 raft_consensus.cc:729] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:29.737960 19088 consensus_queue.cc:260] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:29.738215 19397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99128e8f96dc4e1582ec51cbcb8085e0" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "a2f6fcadc2e047ddb0c111c95420861a"
I20240502 14:15:29.738323 19088 raft_consensus.cc:3037] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:29.738379 19397 raft_consensus.cc:3037] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:29.738770 19846 raft_consensus.cc:979] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f: : Instructing follower 4c60ef9c4e8f439facdfdc6fd669ae78 to start an election
I20240502 14:15:29.738859 19846 raft_consensus.cc:1067] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 4 LEADER]: Signalling peer 4c60ef9c4e8f439facdfdc6fd669ae78 to start an election
I20240502 14:15:29.739372 19160 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1"
dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
from {username='slave'} at 127.0.0.1:49458
I20240502 14:15:29.739547 19160 raft_consensus.cc:482] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:29.739615 19160 raft_consensus.cc:3037] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:29.740880 19088 raft_consensus.cc:2454] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 2.
I20240502 14:15:29.741040 19397 raft_consensus.cc:2454] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 2.
I20240502 14:15:29.741196 18982 leader_election.cc:302] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:29.737951 19787 tablet_bootstrap.cc:492] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66: Bootstrap complete.
I20240502 14:15:29.741591 19787 ts_tablet_manager.cc:1367] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66: Time spent bootstrapping tablet: real 0.074s user 0.032s sys 0.020s
I20240502 14:15:29.742146 19160 raft_consensus.cc:504] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:29.742769 19823 raft_consensus.cc:2781] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:29.742903 19823 raft_consensus.cc:686] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Becoming Leader. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:29.742949 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:29.743096 19088 raft_consensus.cc:3032] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:15:29.743166 19088 raft_consensus.cc:729] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:29.743130 19823 consensus_queue.cc:237] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:29.743410 19088 consensus_queue.cc:260] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:29.743721 19088 raft_consensus.cc:3037] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:29.744494 19160 leader_election.cc:288] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 5 election: Requested vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), d908205271af490dada0a2845191fef4 (127.15.74.70:35089)
I20240502 14:15:29.744858 19549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d01415abc44ec4b52577992892c5b1" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "d908205271af490dada0a2845191fef4"
I20240502 14:15:29.745000 19549 raft_consensus.cc:3037] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:29.747071 19549 raft_consensus.cc:2454] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 5.
I20240502 14:15:29.747413 19131 leader_election.cc:302] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c60ef9c4e8f439facdfdc6fd669ae78, d908205271af490dada0a2845191fef4; no voters:
I20240502 14:15:29.747792 19821 raft_consensus.cc:2781] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:29.749015 19821 raft_consensus.cc:686] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 5 LEADER]: Becoming Leader. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Running, Role: LEADER
I20240502 14:15:29.749197 19088 raft_consensus.cc:2454] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 5.
I20240502 14:15:29.742655 19787 raft_consensus.cc:348] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:29.749368 19821 consensus_queue.cc:237] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:29.749641 19787 raft_consensus.cc:729] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: eac995eafe3e4c388a7caf591d6f1c66, State: Initialized, Role: LEARNER
I20240502 14:15:29.749822 19787 consensus_queue.cc:260] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:29.752768 18910 catalog_manager.cc:5484] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: term changed from 4 to 5, leader changed from 80654f573c764bc18e65d583b751997f (127.15.74.66) to 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67). New cstate: current_term: 5 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:29.752774 18906 catalog_manager.cc:5484] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: term changed from 1 to 2, leader changed from 80654f573c764bc18e65d583b751997f (127.15.74.66) to 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65). New cstate: current_term: 2 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:29.755676 19787 ts_tablet_manager.cc:1398] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240502 14:15:29.756639 19409 tablet_copy_service.cc:342] P a2f6fcadc2e047ddb0c111c95420861a: Request end of tablet copy session eac995eafe3e4c388a7caf591d6f1c66-567ee47202bd46048e318b4dbbf3ff03 received from {username='slave'} at 127.0.0.1:36514
I20240502 14:15:29.756739 19409 tablet_copy_service.cc:433] P a2f6fcadc2e047ddb0c111c95420861a: ending tablet copy session eac995eafe3e4c388a7caf591d6f1c66-567ee47202bd46048e318b4dbbf3ff03 on tablet 567ee47202bd46048e318b4dbbf3ff03 with peer eac995eafe3e4c388a7caf591d6f1c66
I20240502 14:15:29.780838 19804 tablet_bootstrap.cc:492] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=119 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:29.781358 19804 tablet_bootstrap.cc:492] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4: Bootstrap complete.
I20240502 14:15:29.781682 19804 ts_tablet_manager.cc:1367] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4: Time spent bootstrapping tablet: real 0.087s user 0.039s sys 0.000s
I20240502 14:15:29.782444 19804 raft_consensus.cc:348] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:29.782632 19804 raft_consensus.cc:729] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d908205271af490dada0a2845191fef4, State: Initialized, Role: LEARNER
I20240502 14:15:29.782812 19804 consensus_queue.cc:260] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:29.783342 19804 ts_tablet_manager.cc:1398] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:29.783990 19026 tablet_copy_service.cc:342] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Request end of tablet copy session d908205271af490dada0a2845191fef4-f260d59158a344beb686ea0f9746f743 received from {username='slave'} at 127.0.0.1:54696
I20240502 14:15:29.784094 19026 tablet_copy_service.cc:433] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: ending tablet copy session d908205271af490dada0a2845191fef4-f260d59158a344beb686ea0f9746f743 on tablet f260d59158a344beb686ea0f9746f743 with peer d908205271af490dada0a2845191fef4
I20240502 14:15:29.796532 19814 tablet_bootstrap.cc:492] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=109 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:29.796936 19814 tablet_bootstrap.cc:492] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4: Bootstrap complete.
I20240502 14:15:29.797207 19814 ts_tablet_manager.cc:1367] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4: Time spent bootstrapping tablet: real 0.072s user 0.034s sys 0.006s
I20240502 14:15:29.797976 19814 raft_consensus.cc:348] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:29.798218 19814 raft_consensus.cc:729] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d908205271af490dada0a2845191fef4, State: Initialized, Role: LEARNER
I20240502 14:15:29.798408 19814 consensus_queue.cc:260] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:29.798883 19814 ts_tablet_manager.cc:1398] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:29.800307 19485 tablet_copy_service.cc:342] P eac995eafe3e4c388a7caf591d6f1c66: Request end of tablet copy session d908205271af490dada0a2845191fef4-73c203a4764245dba0944537c7d10129 received from {username='slave'} at 127.0.0.1:55232
I20240502 14:15:29.800410 19485 tablet_copy_service.cc:433] P eac995eafe3e4c388a7caf591d6f1c66: ending tablet copy session d908205271af490dada0a2845191fef4-73c203a4764245dba0944537c7d10129 on tablet 73c203a4764245dba0944537c7d10129 with peer d908205271af490dada0a2845191fef4
I20240502 14:15:29.955054 19473 raft_consensus.cc:1203] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.23->[1.24-1.24] Dedup: 1.24->[]
I20240502 14:15:29.975827 19397 raft_consensus.cc:1203] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20240502 14:15:29.984697 19397 raft_consensus.cc:3037] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a [term 2 LEARNER]: Advancing to term 3
I20240502 14:15:30.028923 19840 raft_consensus.cc:979] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: : Instructing follower 4c60ef9c4e8f439facdfdc6fd669ae78 to start an election
I20240502 14:15:30.029098 19840 raft_consensus.cc:1067] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Signalling peer 4c60ef9c4e8f439facdfdc6fd669ae78 to start an election
I20240502 14:15:30.029718 19160 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ef2dcb2456949cfa34b5c2e8b0d2e6d"
dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
from {username='slave'} at 127.0.0.1:49454
I20240502 14:15:30.029922 19160 raft_consensus.cc:482] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:30.030030 19160 raft_consensus.cc:3037] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:30.032477 19160 raft_consensus.cc:504] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:30.033020 19160 leader_election.cc:288] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 2 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:30.033402 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2dcb2456949cfa34b5c2e8b0d2e6d" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:30.033584 19088 raft_consensus.cc:3037] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:30.035919 19088 raft_consensus.cc:2454] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 2.
I20240502 14:15:30.036193 19012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2dcb2456949cfa34b5c2e8b0d2e6d" candidate_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:30.036265 19131 leader_election.cc:302] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c60ef9c4e8f439facdfdc6fd669ae78, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:30.036373 19012 raft_consensus.cc:3032] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:30.036459 19012 raft_consensus.cc:729] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:30.036520 19821 raft_consensus.cc:2781] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:30.036625 19821 raft_consensus.cc:686] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Becoming Leader. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Running, Role: LEADER
I20240502 14:15:30.036727 19012 consensus_queue.cc:260] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:30.036831 19821 consensus_queue.cc:237] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:30.037180 19012 raft_consensus.cc:3037] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:30.039976 18906 catalog_manager.cc:5484] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: term changed from 1 to 2, leader changed from 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) to 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67). New cstate: current_term: 2 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:30.041169 19012 raft_consensus.cc:2454] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c60ef9c4e8f439facdfdc6fd669ae78 in term 2.
I20240502 14:15:30.048678 19160 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ef2dcb2456949cfa34b5c2e8b0d2e6d"
dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49446
I20240502 14:15:30.048828 19160 raft_consensus.cc:595] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Received request to transfer leadership
I20240502 14:15:30.055588 19160 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ef2dcb2456949cfa34b5c2e8b0d2e6d"
dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49446
I20240502 14:15:30.055744 19160 raft_consensus.cc:595] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Received request to transfer leadership
W20240502 14:15:30.056051 18898 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 6ef2dcb2456949cfa34b5c2e8b0d2e6d already in progress [suppressed 1 similar messages]
I20240502 14:15:30.057004 19160 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ef2dcb2456949cfa34b5c2e8b0d2e6d"
dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49446
I20240502 14:15:30.057132 19160 raft_consensus.cc:595] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Received request to transfer leadership
W20240502 14:15:30.057482 18962 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 6ef2dcb2456949cfa34b5c2e8b0d2e6d already in progress
W20240502 14:15:30.057552 18962 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 6ef2dcb2456949cfa34b5c2e8b0d2e6d already in progress
I20240502 14:15:30.107645 19549 raft_consensus.cc:1203] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.26->[2.27-2.27] Dedup: 2.27->[]
I20240502 14:15:30.156780 19088 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P 80654f573c764bc18e65d583b751997f [term 5 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240502 14:15:30.157227 19821 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:15:30.161170 19549 raft_consensus.cc:1261] T 35d01415abc44ec4b52577992892c5b1 P d908205271af490dada0a2845191fef4 [term 5 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240502 14:15:30.162185 19821 consensus_queue.cc:1035] T 35d01415abc44ec4b52577992892c5b1 P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:15:30.185045 19473 raft_consensus.cc:1203] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.31->[4.32-4.32] Dedup: 4.32->[]
I20240502 14:15:30.280774 19549 raft_consensus.cc:1203] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.28->[2.29-2.29] Dedup: 2.29->[]
I20240502 14:15:30.337381 19398 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:15:30.337882 19840 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:30.342718 19090 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:15:30.343159 19867 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:30.497545 19870 raft_consensus.cc:1050] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4: attempting to promote NON_VOTER a2f6fcadc2e047ddb0c111c95420861a to VOTER
I20240502 14:15:30.498320 19870 consensus_queue.cc:237] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:30.500310 19398 raft_consensus.cc:1261] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a [term 2 LEARNER]: Refusing update from remote peer d908205271af490dada0a2845191fef4: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 14:15:30.500762 19090 raft_consensus.cc:1261] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer d908205271af490dada0a2845191fef4: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 14:15:30.500912 19160 raft_consensus.cc:1261] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Refusing update from remote peer d908205271af490dada0a2845191fef4: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 14:15:30.501349 19869 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:15:30.501828 19870 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:15:30.502076 19872 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:15:30.505398 19869 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:30.505975 19160 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:30.507619 19398 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:30.510550 18906 catalog_manager.cc:5484] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 reported cstate change: config changed from index 30 to 31, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d908205271af490dada0a2845191fef4" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:30.511685 19090 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:30.515527 19549 consensus_queue.cc:237] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:30.517292 19398 raft_consensus.cc:1261] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a [term 2 FOLLOWER]: Refusing update from remote peer d908205271af490dada0a2845191fef4: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 14:15:30.517815 19160 raft_consensus.cc:1261] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Refusing update from remote peer d908205271af490dada0a2845191fef4: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 14:15:30.518014 19839 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:15:30.518766 19839 consensus_queue.cc:1035] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:15:30.520190 19872 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:30.520417 19398 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P a2f6fcadc2e047ddb0c111c95420861a [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:30.524036 18961 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 76a8747fc61249759eb84a0110df4efb with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:30.523938 19160 raft_consensus.cc:2932] T 76a8747fc61249759eb84a0110df4efb P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:30.527631 18911 catalog_manager.cc:5484] T 76a8747fc61249759eb84a0110df4efb P d908205271af490dada0a2845191fef4 reported cstate change: config changed from index 31 to 32, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New cstate: current_term: 2 leader_uuid: "d908205271af490dada0a2845191fef4" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:30.530897 19080 tablet_service.cc:1514] Processing DeleteTablet for tablet 76a8747fc61249759eb84a0110df4efb with delete_type TABLET_DATA_TOMBSTONED (TS 80654f573c764bc18e65d583b751997f not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:60996
I20240502 14:15:30.532892 19876 tablet_replica.cc:330] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f: stopping tablet replica
I20240502 14:15:30.533099 19876 raft_consensus.cc:2229] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:30.533386 19876 raft_consensus.cc:2258] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:30.534300 19876 ts_tablet_manager.cc:1875] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:30.539177 19876 ts_tablet_manager.cc:1888] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240502 14:15:30.539290 19876 log.cc:1196] T 76a8747fc61249759eb84a0110df4efb P 80654f573c764bc18e65d583b751997f: Deleting WAL directory at /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-1-root/wals/76a8747fc61249759eb84a0110df4efb
I20240502 14:15:30.540009 18901 catalog_manager.cc:4830] TS 80654f573c764bc18e65d583b751997f (127.15.74.66:44583): tablet 76a8747fc61249759eb84a0110df4efb (table test-workload [id=682791e08deb409e86af13a3a7eb1967]) successfully deleted
I20240502 14:15:30.544456 19012 raft_consensus.cc:1261] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240502 14:15:30.544893 19848 consensus_queue.cc:1035] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 14:15:30.547005 19848 raft_consensus.cc:979] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78: : Instructing follower 1880cd4d2b8e4ec6b42dfd8d2660f5d7 to start an election
I20240502 14:15:30.547097 19848 raft_consensus.cc:1067] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Signalling peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7 to start an election
I20240502 14:15:30.548416 19090 raft_consensus.cc:1261] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240502 14:15:30.548463 19011 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ef2dcb2456949cfa34b5c2e8b0d2e6d"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
from {username='slave'} at 127.0.0.1:54284
I20240502 14:15:30.548635 19011 raft_consensus.cc:482] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:30.548705 19472 raft_consensus.cc:3037] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66 [term 1 LEARNER]: Advancing to term 2
I20240502 14:15:30.550783 19880 raft_consensus.cc:1050] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f: attempting to promote NON_VOTER a2f6fcadc2e047ddb0c111c95420861a to VOTER
I20240502 14:15:30.551529 19821 consensus_queue.cc:1035] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 14:15:30.551399 19880 consensus_queue.cc:237] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 27, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:30.551620 19472 raft_consensus.cc:1261] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66 [term 2 LEARNER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240502 14:15:30.552135 19821 consensus_queue.cc:1035] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 14:15:30.548736 19011 raft_consensus.cc:3037] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:30.558121 19011 raft_consensus.cc:504] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:30.558559 19012 raft_consensus.cc:1226] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Rejecting Update request from peer 4c60ef9c4e8f439facdfdc6fd669ae78 for earlier term 2. Current term is 3. Ops: []
I20240502 14:15:30.559417 19821 consensus_queue.cc:1046] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: INVALID_TERM, Last received: 2.25, Next index: 26, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 14:15:30.559741 19821 raft_consensus.cc:3032] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:30.559829 19821 raft_consensus.cc:729] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Running, Role: LEADER
I20240502 14:15:30.560120 19821 consensus_queue.cc:260] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:30.560444 19821 raft_consensus.cc:3037] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:30.568393 19090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2dcb2456949cfa34b5c2e8b0d2e6d" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:30.568578 19090 raft_consensus.cc:3037] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:30.569011 19160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2dcb2456949cfa34b5c2e8b0d2e6d" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:30.571118 19090 raft_consensus.cc:2454] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 3.
I20240502 14:15:30.571532 19160 raft_consensus.cc:2454] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 3.
I20240502 14:15:30.571894 18982 leader_election.cc:302] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:30.572225 19867 raft_consensus.cc:2781] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:30.572345 19867 raft_consensus.cc:686] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 LEADER]: Becoming Leader. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:30.572595 19867 consensus_queue.cc:237] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 2.25, Last appended by leader: 25, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:30.574280 19011 leader_election.cc:288] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 3 election: Requested vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:30.575023 18906 catalog_manager.cc:5484] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: term changed from 2 to 3, leader changed from 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) to 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65). New cstate: current_term: 3 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:30.575742 19160 raft_consensus.cc:1261] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 14:15:30.576030 19398 raft_consensus.cc:1261] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a [term 3 LEARNER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 14:15:30.576498 19880 consensus_queue.cc:1035] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:30.576727 19012 raft_consensus.cc:1261] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 14:15:30.577404 19880 consensus_queue.cc:1035] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:30.577879 19879 consensus_queue.cc:1035] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:30.579806 19828 raft_consensus.cc:2932] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 27 to 29, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:30.580703 19012 raft_consensus.cc:2932] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:30.583138 19160 raft_consensus.cc:2932] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:30.583495 19397 raft_consensus.cc:2932] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:30.584803 18911 catalog_manager.cc:5484] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f reported cstate change: config changed from index 27 to 29, a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:30.588356 19867 raft_consensus.cc:1050] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: attempting to promote NON_VOTER d908205271af490dada0a2845191fef4 to VOTER
I20240502 14:15:30.589080 19867 consensus_queue.cc:237] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 25, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:30.589521 19090 consensus_queue.cc:237] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:30.590922 19397 raft_consensus.cc:1261] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a [term 3 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:15:30.590942 19011 raft_consensus.cc:1261] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:15:30.591351 19880 consensus_queue.cc:1035] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:30.591627 19828 consensus_queue.cc:1035] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:30.593897 19880 raft_consensus.cc:2932] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:30.594796 19011 raft_consensus.cc:2932] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:30.602734 19549 raft_consensus.cc:1261] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [term 2 LEARNER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 14:15:30.603292 19867 consensus_queue.cc:1035] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:30.603767 19160 raft_consensus.cc:1261] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 14:15:30.604287 18911 catalog_manager.cc:5484] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: config changed from index 29 to 30, VOTER 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) evicted. New cstate: current_term: 3 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:30.604764 19823 consensus_queue.cc:1035] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:30.605942 19088 raft_consensus.cc:1261] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 14:15:30.606689 19823 consensus_queue.cc:1035] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:30.609558 19155 tablet_service.cc:1514] Processing DeleteTablet for tablet 154f396806954d2eb24baa4a86ba876d with delete_type TABLET_DATA_TOMBSTONED (TS 4c60ef9c4e8f439facdfdc6fd669ae78 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:49446
I20240502 14:15:30.609889 19831 raft_consensus.cc:1050] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a: attempting to promote NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 to VOTER
I20240502 14:15:30.610878 19893 tablet_replica.cc:330] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78: stopping tablet replica
I20240502 14:15:30.610787 19831 consensus_queue.cc:237] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 30, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
I20240502 14:15:30.612919 19159 raft_consensus.cc:2932] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.611126 19893 raft_consensus.cc:2229] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:30.612082 19867 raft_consensus.cc:2932] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.613615 19893 raft_consensus.cc:2258] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:30.614990 19397 raft_consensus.cc:2932] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } }
I20240502 14:15:30.617012 18901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 154f396806954d2eb24baa4a86ba876d with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:30.618273 18911 catalog_manager.cc:5484] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: config changed from index 27 to 28, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.621206 19893 ts_tablet_manager.cc:1875] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:30.621752 19088 raft_consensus.cc:2932] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.617412 19549 raft_consensus.cc:2932] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.627851 19012 consensus_queue.cc:237] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:30.630923 19549 raft_consensus.cc:1261] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 14:15:30.632054 19823 consensus_queue.cc:1035] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:30.632442 19159 raft_consensus.cc:1261] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 14:15:30.633179 19823 consensus_queue.cc:1035] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:30.634269 19840 raft_consensus.cc:2932] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.634797 19549 raft_consensus.cc:2932] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.635886 19159 raft_consensus.cc:2932] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.638197 18900 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f260d59158a344beb686ea0f9746f743 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:30.639573 19012 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240502 14:15:30.639403 18911 catalog_manager.cc:5484] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 reported cstate change: config changed from index 28 to 29, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New cstate: current_term: 2 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.640758 19888 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.001s
I20240502 14:15:30.642122 19160 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240502 14:15:30.644341 19472 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66 [term 4 LEARNER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240502 14:15:30.644388 19080 tablet_service.cc:1514] Processing DeleteTablet for tablet f260d59158a344beb686ea0f9746f743 with delete_type TABLET_DATA_TOMBSTONED (TS 80654f573c764bc18e65d583b751997f not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:60996
I20240502 14:15:30.644703 19876 tablet_replica.cc:330] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f: stopping tablet replica
I20240502 14:15:30.644896 19876 raft_consensus.cc:2229] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:30.645084 19876 raft_consensus.cc:2258] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:30.645666 19888 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:15:30.646176 19876 ts_tablet_manager.cc:1875] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:30.647873 19888 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:15:30.651145 19892 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 32 to 33, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:30.653738 19876 ts_tablet_manager.cc:1888] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.28
I20240502 14:15:30.653841 19876 log.cc:1196] T f260d59158a344beb686ea0f9746f743 P 80654f573c764bc18e65d583b751997f: Deleting WAL directory at /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-1-root/wals/f260d59158a344beb686ea0f9746f743
I20240502 14:15:30.642832 19893 ts_tablet_manager.cc:1888] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.29
I20240502 14:15:30.654430 18901 catalog_manager.cc:4830] TS 80654f573c764bc18e65d583b751997f (127.15.74.66:44583): tablet f260d59158a344beb686ea0f9746f743 (table test-workload [id=682791e08deb409e86af13a3a7eb1967]) successfully deleted
I20240502 14:15:30.654444 19893 log.cc:1196] T 154f396806954d2eb24baa4a86ba876d P 4c60ef9c4e8f439facdfdc6fd669ae78: Deleting WAL directory at /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-2-root/wals/154f396806954d2eb24baa4a86ba876d
I20240502 14:15:30.655249 18898 catalog_manager.cc:4830] TS 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767): tablet 154f396806954d2eb24baa4a86ba876d (table test-workload [id=682791e08deb409e86af13a3a7eb1967]) successfully deleted
I20240502 14:15:30.656596 19160 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:30.657232 18906 catalog_manager.cc:5484] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a reported cstate change: config changed from index 32 to 33, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "a2f6fcadc2e047ddb0c111c95420861a" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:30.656139 19472 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:30.663462 19397 consensus_queue.cc:237] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
I20240502 14:15:30.663488 19011 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:30.665776 19160 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240502 14:15:30.666347 19472 raft_consensus.cc:1261] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66 [term 4 FOLLOWER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240502 14:15:30.666802 19831 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 14:15:30.667300 19892 consensus_queue.cc:1035] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 14:15:30.669689 19831 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P a2f6fcadc2e047ddb0c111c95420861a [term 4 LEADER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:30.670828 19160 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:30.672044 19474 raft_consensus.cc:2932] T 567ee47202bd46048e318b4dbbf3ff03 P eac995eafe3e4c388a7caf591d6f1c66 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:30.672956 18961 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 567ee47202bd46048e318b4dbbf3ff03 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:30.676355 18910 catalog_manager.cc:5484] T 567ee47202bd46048e318b4dbbf3ff03 P 4c60ef9c4e8f439facdfdc6fd669ae78 reported cstate change: config changed from index 33 to 34, VOTER 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) evicted. New cstate: current_term: 4 leader_uuid: "a2f6fcadc2e047ddb0c111c95420861a" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:30.679723 19005 tablet_service.cc:1514] Processing DeleteTablet for tablet 567ee47202bd46048e318b4dbbf3ff03 with delete_type TABLET_DATA_TOMBSTONED (TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:54276
I20240502 14:15:30.680492 19899 tablet_replica.cc:330] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: stopping tablet replica
I20240502 14:15:30.680703 19899 raft_consensus.cc:2229] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:30.680907 19899 raft_consensus.cc:2258] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:30.681708 19899 ts_tablet_manager.cc:1875] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:30.687793 19899 ts_tablet_manager.cc:1888] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.33
I20240502 14:15:30.687880 19899 log.cc:1196] T 567ee47202bd46048e318b4dbbf3ff03 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Deleting WAL directory at /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-0-root/wals/567ee47202bd46048e318b4dbbf3ff03
I20240502 14:15:30.688550 18900 catalog_manager.cc:4830] TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141): tablet 567ee47202bd46048e318b4dbbf3ff03 (table test-workload [id=682791e08deb409e86af13a3a7eb1967]) successfully deleted
I20240502 14:15:30.890523 19833 raft_consensus.cc:1050] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66: attempting to promote NON_VOTER d908205271af490dada0a2845191fef4 to VOTER
I20240502 14:15:30.891350 19833 consensus_queue.cc:237] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 27, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:30.893585 19088 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer eac995eafe3e4c388a7caf591d6f1c66: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:15:30.893823 19011 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Refusing update from remote peer eac995eafe3e4c388a7caf591d6f1c66: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:15:30.894219 19900 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:30.894477 19549 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4 [term 2 LEARNER]: Refusing update from remote peer eac995eafe3e4c388a7caf591d6f1c66: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 14:15:30.894749 19903 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:30.895098 19833 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:30.901086 19833 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.901798 19549 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.904848 19011 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.905323 18906 catalog_manager.cc:5484] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 reported cstate change: config changed from index 29 to 30, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "eac995eafe3e4c388a7caf591d6f1c66" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:30.933837 19090 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, d908205271af490dada0a2845191fef4 (127.15.74.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.941977 19474 consensus_queue.cc:237] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:30.943670 19549 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4 [term 2 FOLLOWER]: Refusing update from remote peer eac995eafe3e4c388a7caf591d6f1c66: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 14:15:30.944185 19088 raft_consensus.cc:1261] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer eac995eafe3e4c388a7caf591d6f1c66: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 14:15:30.944515 19903 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:15:30.944794 19833 consensus_queue.cc:1035] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:15:30.947324 19901 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P eac995eafe3e4c388a7caf591d6f1c66 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.947926 19088 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.950536 18961 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 73c203a4764245dba0944537c7d10129 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:30.951915 18906 catalog_manager.cc:5484] T 73c203a4764245dba0944537c7d10129 P 80654f573c764bc18e65d583b751997f reported cstate change: config changed from index 30 to 31, VOTER 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) evicted. New cstate: current_term: 2 leader_uuid: "eac995eafe3e4c388a7caf591d6f1c66" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.953367 19549 raft_consensus.cc:2932] T 73c203a4764245dba0944537c7d10129 P d908205271af490dada0a2845191fef4 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } }
I20240502 14:15:30.955919 19005 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c203a4764245dba0944537c7d10129 with delete_type TABLET_DATA_TOMBSTONED (TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:54276
I20240502 14:15:30.956302 19899 tablet_replica.cc:330] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: stopping tablet replica
I20240502 14:15:30.956558 19899 raft_consensus.cc:2229] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:30.956812 19899 raft_consensus.cc:2258] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:30.958688 19899 ts_tablet_manager.cc:1875] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:30.964578 19474 raft_consensus.cc:3037] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66 [term 2 LEARNER]: Advancing to term 3
I20240502 14:15:30.966846 19474 raft_consensus.cc:1261] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66 [term 3 LEARNER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
I20240502 14:15:30.967319 19823 consensus_queue.cc:1035] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:30.969153 19899 ts_tablet_manager.cc:1888] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.30
I20240502 14:15:30.969254 19823 raft_consensus.cc:1050] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: attempting to promote NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 to VOTER
I20240502 14:15:30.969301 19899 log.cc:1196] T 73c203a4764245dba0944537c7d10129 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Deleting WAL directory at /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-0-root/wals/73c203a4764245dba0944537c7d10129
W20240502 14:15:30.969358 19823 raft_consensus.cc:1054] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Unable to promote non-voter eac995eafe3e4c388a7caf591d6f1c66: Illegal state: Leader has not yet committed an operation in its own term
I20240502 14:15:30.970196 18900 catalog_manager.cc:4830] TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141): tablet 73c203a4764245dba0944537c7d10129 (table test-workload [id=682791e08deb409e86af13a3a7eb1967]) successfully deleted
I20240502 14:15:30.981317 19088 raft_consensus.cc:1261] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
I20240502 14:15:30.981814 19823 consensus_queue.cc:1035] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:30.986670 19160 raft_consensus.cc:1261] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
I20240502 14:15:30.987221 19867 consensus_queue.cc:1035] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:30.996099 19840 raft_consensus.cc:1050] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: attempting to promote NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 to VOTER
I20240502 14:15:30.996752 19840 consensus_queue.cc:237] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 3.26, Last appended by leader: 25, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
I20240502 14:15:30.998674 19088 raft_consensus.cc:1261] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240502 14:15:30.998849 19160 raft_consensus.cc:1261] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240502 14:15:30.999202 19474 raft_consensus.cc:1261] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66 [term 3 LEARNER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240502 14:15:30.999486 19910 consensus_queue.cc:1035] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:31.000089 19910 consensus_queue.cc:1035] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:31.000281 19823 consensus_queue.cc:1035] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:31.003063 19867 raft_consensus.cc:2932] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 LEADER]: Committing config change with OpId 3.27: config changed from index 24 to 27, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:31.003906 19088 raft_consensus.cc:2932] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Committing config change with OpId 3.27: config changed from index 24 to 27, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:31.005617 19160 raft_consensus.cc:2932] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Committing config change with OpId 3.27: config changed from index 24 to 27, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:31.008852 18905 catalog_manager.cc:5484] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f reported cstate change: config changed from index 24 to 27, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:31.012632 19474 raft_consensus.cc:2932] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66 [term 3 FOLLOWER]: Committing config change with OpId 3.27: config changed from index 24 to 27, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:31.017657 19011 consensus_queue.cc:237] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 25, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
I20240502 14:15:31.019348 19474 raft_consensus.cc:1261] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66 [term 3 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 14:15:31.019853 19088 raft_consensus.cc:1261] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 14:15:31.020570 19910 consensus_queue.cc:1035] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:31.021090 19840 consensus_queue.cc:1035] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:31.022946 19867 raft_consensus.cc:2932] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 LEADER]: Committing config change with OpId 3.28: config changed from index 27 to 28, VOTER 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:31.027810 18900 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6ef2dcb2456949cfa34b5c2e8b0d2e6d with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:31.027858 19474 raft_consensus.cc:2932] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P eac995eafe3e4c388a7caf591d6f1c66 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 27 to 28, VOTER 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:31.029739 18905 catalog_manager.cc:5484] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: config changed from index 27 to 28, VOTER 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) evicted. New cstate: current_term: 3 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:31.031193 19088 raft_consensus.cc:2932] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 27 to 28, VOTER 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:31.035913 19155 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ef2dcb2456949cfa34b5c2e8b0d2e6d with delete_type TABLET_DATA_TOMBSTONED (TS 4c60ef9c4e8f439facdfdc6fd669ae78 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:49446
I20240502 14:15:31.036388 19893 tablet_replica.cc:330] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78: stopping tablet replica
I20240502 14:15:31.036625 19893 raft_consensus.cc:2229] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:31.036999 19893 raft_consensus.cc:2258] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:31.038529 19893 ts_tablet_manager.cc:1875] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:31.047454 19893 ts_tablet_manager.cc:1888] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.27
I20240502 14:15:31.047597 19893 log.cc:1196] T 6ef2dcb2456949cfa34b5c2e8b0d2e6d P 4c60ef9c4e8f439facdfdc6fd669ae78: Deleting WAL directory at /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-2-root/wals/6ef2dcb2456949cfa34b5c2e8b0d2e6d
I20240502 14:15:31.050961 18898 catalog_manager.cc:4830] TS 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767): tablet 6ef2dcb2456949cfa34b5c2e8b0d2e6d (table test-workload [id=682791e08deb409e86af13a3a7eb1967]) successfully deleted
I20240502 14:15:31.674001 18967 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:31.677084 19011 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
mode: GRACEFUL
new_leader_uuid: "80654f573c764bc18e65d583b751997f"
from {username='slave'} at 127.0.0.1:54318
I20240502 14:15:31.677266 19011 raft_consensus.cc:595] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 LEADER]: Received request to transfer leadership to TS 80654f573c764bc18e65d583b751997f
I20240502 14:15:31.678242 19160 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "342705e9da8043ae9aebe50310650d2c"
dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
mode: GRACEFUL
new_leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
from {username='slave'} at 127.0.0.1:49498
I20240502 14:15:31.678391 19160 raft_consensus.cc:595] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Received request to transfer leadership to TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7
I20240502 14:15:31.679361 19011 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f260d59158a344beb686ea0f9746f743"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
mode: GRACEFUL
new_leader_uuid: "d908205271af490dada0a2845191fef4"
from {username='slave'} at 127.0.0.1:54318
I20240502 14:15:31.679543 19011 raft_consensus.cc:595] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Received request to transfer leadership to TS d908205271af490dada0a2845191fef4
I20240502 14:15:31.679919 18967 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 14:15:31.680135 18967 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:31.696239 19840 raft_consensus.cc:979] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: : Instructing follower d908205271af490dada0a2845191fef4 to start an election
I20240502 14:15:31.696382 19840 raft_consensus.cc:1067] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Signalling peer d908205271af490dada0a2845191fef4 to start an election
I20240502 14:15:31.696965 19549 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f260d59158a344beb686ea0f9746f743"
dest_uuid: "d908205271af490dada0a2845191fef4"
from {username='slave'} at 127.0.0.1:49186
I20240502 14:15:31.697119 19549 raft_consensus.cc:482] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:31.697187 19549 raft_consensus.cc:3037] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:31.699205 19549 raft_consensus.cc:504] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:31.699894 19549 leader_election.cc:288] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [CANDIDATE]: Term 3 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:31.700342 19160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f260d59158a344beb686ea0f9746f743" candidate_uuid: "d908205271af490dada0a2845191fef4" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:31.700304 19011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f260d59158a344beb686ea0f9746f743" candidate_uuid: "d908205271af490dada0a2845191fef4" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:31.700557 19160 raft_consensus.cc:3037] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:31.700595 19011 raft_consensus.cc:3032] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:31.700664 19011 raft_consensus.cc:729] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:31.700897 19011 consensus_queue.cc:260] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:31.701182 19011 raft_consensus.cc:3037] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:31.702736 19160 raft_consensus.cc:2454] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d908205271af490dada0a2845191fef4 in term 3.
I20240502 14:15:31.703163 19011 raft_consensus.cc:2454] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d908205271af490dada0a2845191fef4 in term 3.
I20240502 14:15:31.703135 19512 leader_election.cc:302] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c60ef9c4e8f439facdfdc6fd669ae78, d908205271af490dada0a2845191fef4; no voters:
I20240502 14:15:31.703454 19872 raft_consensus.cc:2781] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:31.703895 19872 raft_consensus.cc:686] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [term 3 LEADER]: Becoming Leader. State: Replica: d908205271af490dada0a2845191fef4, State: Running, Role: LEADER
I20240502 14:15:31.704150 19911 raft_consensus.cc:979] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: : Instructing follower 80654f573c764bc18e65d583b751997f to start an election
I20240502 14:15:31.704265 19840 raft_consensus.cc:1067] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 LEADER]: Signalling peer 80654f573c764bc18e65d583b751997f to start an election
I20240502 14:15:31.704195 19872 consensus_queue.cc:237] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:31.704845 19088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b"
dest_uuid: "80654f573c764bc18e65d583b751997f"
from {username='slave'} at 127.0.0.1:60998
I20240502 14:15:31.704993 19088 raft_consensus.cc:482] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:31.705088 19088 raft_consensus.cc:3037] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:31.706986 19088 raft_consensus.cc:504] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:31.706976 18905 catalog_manager.cc:5484] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 reported cstate change: term changed from 2 to 3, leader changed from 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) to d908205271af490dada0a2845191fef4 (127.15.74.70). New cstate: current_term: 3 leader_uuid: "d908205271af490dada0a2845191fef4" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:31.707777 19088 leader_election.cc:288] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 4 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:31.707909 19011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:31.708089 19011 raft_consensus.cc:3032] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:31.708144 19160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb757082b084620af257fb553d2dd4b" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:31.708179 19011 raft_consensus.cc:729] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:31.708357 19160 raft_consensus.cc:3037] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:31.708591 19011 consensus_queue.cc:260] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:31.709056 19011 raft_consensus.cc:3037] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:31.710222 19160 raft_consensus.cc:2454] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 4.
I20240502 14:15:31.710610 19054 leader_election.cc:302] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c60ef9c4e8f439facdfdc6fd669ae78, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:31.710898 19894 raft_consensus.cc:2781] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:31.711293 19894 raft_consensus.cc:686] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 4 LEADER]: Becoming Leader. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:31.711601 19894 consensus_queue.cc:237] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:31.711922 19011 raft_consensus.cc:2454] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 4.
I20240502 14:15:31.714411 18906 catalog_manager.cc:5484] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f reported cstate change: term changed from 3 to 4, leader changed from 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) to 80654f573c764bc18e65d583b751997f (127.15.74.66). New cstate: current_term: 4 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:31.833508 19919 raft_consensus.cc:979] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78: : Instructing follower 1880cd4d2b8e4ec6b42dfd8d2660f5d7 to start an election
I20240502 14:15:31.833649 19919 raft_consensus.cc:1067] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Signalling peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7 to start an election
I20240502 14:15:31.834275 19011 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "342705e9da8043ae9aebe50310650d2c"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
from {username='slave'} at 127.0.0.1:54284
I20240502 14:15:31.834458 19011 raft_consensus.cc:482] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:31.834568 19011 raft_consensus.cc:3037] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:31.836527 19011 raft_consensus.cc:504] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:31.837061 19011 leader_election.cc:288] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 2 election: Requested vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:31.837476 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342705e9da8043ae9aebe50310650d2c" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:31.837476 19160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342705e9da8043ae9aebe50310650d2c" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78"
I20240502 14:15:31.837702 19088 raft_consensus.cc:3037] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:31.837702 19160 raft_consensus.cc:3032] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:31.837796 19160 raft_consensus.cc:729] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c60ef9c4e8f439facdfdc6fd669ae78, State: Running, Role: LEADER
I20240502 14:15:31.838032 19160 consensus_queue.cc:260] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:31.838378 19160 raft_consensus.cc:3037] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:31.840510 19088 raft_consensus.cc:2454] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 2.
I20240502 14:15:31.840924 18982 leader_election.cc:302] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:31.841084 19160 raft_consensus.cc:2454] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 2.
I20240502 14:15:31.841233 19840 raft_consensus.cc:2781] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:31.841378 19840 raft_consensus.cc:686] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Becoming Leader. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:31.841621 19840 consensus_queue.cc:237] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
I20240502 14:15:31.844126 18906 catalog_manager.cc:5484] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: term changed from 1 to 2, leader changed from 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67) to 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65). New cstate: current_term: 2 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:31.958951 19011 consensus_queue.cc:237] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:31.961163 19398 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 14:15:31.961470 19088 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 14:15:31.961619 19840 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240502 14:15:31.961896 18982 consensus_peers.cc:485] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 -> Peer eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69:44877): Couldn't send request to peer eac995eafe3e4c388a7caf591d6f1c66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99128e8f96dc4e1582ec51cbcb8085e0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:31.962163 19840 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:31.963932 19911 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 26 to 28, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
I20240502 14:15:31.964478 19398 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 26 to 28, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
W20240502 14:15:31.968420 18962 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20240502 14:15:31.968812 18906 catalog_manager.cc:5484] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: config changed from index 26 to 28, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New cstate: current_term: 2 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:31.970276 19088 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 26 to 28, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
W20240502 14:15:31.971614 18962 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 1cb757082b084620af257fb553d2dd4b, TS 4c60ef9c4e8f439facdfdc6fd669ae78 -> TS a2f6fcadc2e047ddb0c111c95420861a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
W20240502 14:15:31.971683 18962 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 1cb757082b084620af257fb553d2dd4b, TS 4c60ef9c4e8f439facdfdc6fd669ae78 -> TS a2f6fcadc2e047ddb0c111c95420861a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } }
W20240502 14:15:31.975932 18962 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 154f396806954d2eb24baa4a86ba876d, TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7 -> TS d908205271af490dada0a2845191fef4 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
W20240502 14:15:31.976004 18962 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 154f396806954d2eb24baa4a86ba876d, TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7 -> TS d908205271af490dada0a2845191fef4 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
W20240502 14:15:31.978240 18962 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 567ee47202bd46048e318b4dbbf3ff03, TS 4c60ef9c4e8f439facdfdc6fd669ae78 -> TS d908205271af490dada0a2845191fef4 move failed, destination replica disappeared from tablet's Raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
W20240502 14:15:31.978302 18962 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 567ee47202bd46048e318b4dbbf3ff03, TS 4c60ef9c4e8f439facdfdc6fd669ae78 -> TS d908205271af490dada0a2845191fef4 move failed, destination replica disappeared from tablet's Raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
W20240502 14:15:31.980417 18962 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 73c203a4764245dba0944537c7d10129, TS 80654f573c764bc18e65d583b751997f -> TS a2f6fcadc2e047ddb0c111c95420861a move failed, destination replica disappeared from tablet's Raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
W20240502 14:15:31.980482 18962 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 73c203a4764245dba0944537c7d10129, TS 80654f573c764bc18e65d583b751997f -> TS a2f6fcadc2e047ddb0c111c95420861a move failed, destination replica disappeared from tablet's Raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
W20240502 14:15:31.982645 18962 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f260d59158a344beb686ea0f9746f743, TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7 -> TS eac995eafe3e4c388a7caf591d6f1c66 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
W20240502 14:15:31.982705 18962 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f260d59158a344beb686ea0f9746f743, TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7 -> TS eac995eafe3e4c388a7caf591d6f1c66 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } }
I20240502 14:15:32.116210 19011 raft_consensus.cc:1261] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Refusing update from remote peer d908205271af490dada0a2845191fef4: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:15:32.116760 19872 consensus_queue.cc:1035] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:32.120937 19160 raft_consensus.cc:1261] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Refusing update from remote peer d908205271af490dada0a2845191fef4: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:15:32.121526 19912 consensus_queue.cc:1035] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:32.179949 19160 raft_consensus.cc:1261] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240502 14:15:32.180457 19894 consensus_queue.cc:1035] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:32.194065 19012 raft_consensus.cc:1261] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240502 14:15:32.194681 19894 consensus_queue.cc:1035] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 14:15:32.234613 19160 raft_consensus.cc:1261] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 14:15:32.235153 19911 consensus_queue.cc:1035] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:32.239900 19088 raft_consensus.cc:1261] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 14:15:32.240423 19922 consensus_queue.cc:1035] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 14:15:32.438879 19937 ts_tablet_manager.cc:902] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66: Initiating tablet copy from peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141)
I20240502 14:15:32.440651 19937 tablet_copy_client.cc:308] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.65:41141
I20240502 14:15:32.441486 19026 tablet_copy_service.cc:140] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Received BeginTabletCopySession request for tablet 99128e8f96dc4e1582ec51cbcb8085e0 from peer eac995eafe3e4c388a7caf591d6f1c66 ({username='slave'} at 127.0.0.1:54324)
I20240502 14:15:32.441632 19026 tablet_copy_service.cc:161] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Beginning new tablet copy session on tablet 99128e8f96dc4e1582ec51cbcb8085e0 from peer eac995eafe3e4c388a7caf591d6f1c66 at {username='slave'} at 127.0.0.1:54324: session id = eac995eafe3e4c388a7caf591d6f1c66-99128e8f96dc4e1582ec51cbcb8085e0
I20240502 14:15:32.444370 19026 tablet_copy_source_session.cc:205] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:32.445461 19937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99128e8f96dc4e1582ec51cbcb8085e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:32.450760 19937 tablet_copy_client.cc:783] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:32.450994 19937 tablet_copy_client.cc:647] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:32.452487 19942 tablet_copy_client.cc:1002] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Injecting latency into file download: 2000
I20240502 14:15:33.680466 18967 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:33.683980 19088 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d"
dest_uuid: "80654f573c764bc18e65d583b751997f"
mode: GRACEFUL
new_leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
from {username='slave'} at 127.0.0.1:34074
I20240502 14:15:33.684206 19088 raft_consensus.cc:595] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 3 LEADER]: Received request to transfer leadership to TS 1880cd4d2b8e4ec6b42dfd8d2660f5d7
I20240502 14:15:33.685165 19012 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99128e8f96dc4e1582ec51cbcb8085e0"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
mode: GRACEFUL
new_leader_uuid: "a2f6fcadc2e047ddb0c111c95420861a"
from {username='slave'} at 127.0.0.1:54318
I20240502 14:15:33.685350 19012 raft_consensus.cc:595] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Received request to transfer leadership to TS a2f6fcadc2e047ddb0c111c95420861a
I20240502 14:15:33.685773 18967 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 14:15:33.685992 18967 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:34.090513 19950 raft_consensus.cc:979] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f: : Instructing follower 1880cd4d2b8e4ec6b42dfd8d2660f5d7 to start an election
I20240502 14:15:34.090781 19915 raft_consensus.cc:1067] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 3 LEADER]: Signalling peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7 to start an election
I20240502 14:15:34.091348 19011 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
from {username='slave'} at 127.0.0.1:54300
I20240502 14:15:34.091596 19011 raft_consensus.cc:482] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:34.091706 19011 raft_consensus.cc:3037] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:34.093998 19011 raft_consensus.cc:504] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:34.094730 19011 leader_election.cc:288] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 4 election: Requested vote from peers 80654f573c764bc18e65d583b751997f (127.15.74.66:44583), a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68:41443)
I20240502 14:15:34.095059 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:34.095232 19088 raft_consensus.cc:3032] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:34.095312 19088 raft_consensus.cc:729] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:34.095564 19088 consensus_queue.cc:260] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:34.095827 19088 raft_consensus.cc:3037] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:34.097484 19088 raft_consensus.cc:2454] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 4.
I20240502 14:15:34.095139 19398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d" candidate_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "a2f6fcadc2e047ddb0c111c95420861a"
I20240502 14:15:34.097924 19398 raft_consensus.cc:3037] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:34.097985 18982 leader_election.cc:302] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, 80654f573c764bc18e65d583b751997f; no voters:
I20240502 14:15:34.098294 19947 raft_consensus.cc:2781] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:34.098420 19947 raft_consensus.cc:686] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 LEADER]: Becoming Leader. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:34.098677 19947 consensus_queue.cc:237] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:34.100703 19398 raft_consensus.cc:2454] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1880cd4d2b8e4ec6b42dfd8d2660f5d7 in term 4.
I20240502 14:15:34.102267 19946 raft_consensus.cc:979] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: : Instructing follower a2f6fcadc2e047ddb0c111c95420861a to start an election
I20240502 14:15:34.102409 19946 raft_consensus.cc:1067] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Signalling peer a2f6fcadc2e047ddb0c111c95420861a to start an election
I20240502 14:15:34.103538 19397 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99128e8f96dc4e1582ec51cbcb8085e0"
dest_uuid: "a2f6fcadc2e047ddb0c111c95420861a"
from {username='slave'} at 127.0.0.1:44214
I20240502 14:15:34.103688 19397 raft_consensus.cc:482] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:34.103780 19397 raft_consensus.cc:3037] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:34.103958 18906 catalog_manager.cc:5484] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: term changed from 3 to 4, leader changed from 80654f573c764bc18e65d583b751997f (127.15.74.66) to 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65). New cstate: current_term: 4 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:34.105707 19397 raft_consensus.cc:504] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:34.106329 19397 leader_election.cc:288] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [CANDIDATE]: Term 3 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:34.106683 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99128e8f96dc4e1582ec51cbcb8085e0" candidate_uuid: "a2f6fcadc2e047ddb0c111c95420861a" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "80654f573c764bc18e65d583b751997f"
I20240502 14:15:34.106848 19088 raft_consensus.cc:3037] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:34.108498 19088 raft_consensus.cc:2454] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f6fcadc2e047ddb0c111c95420861a in term 3.
I20240502 14:15:34.108983 19012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99128e8f96dc4e1582ec51cbcb8085e0" candidate_uuid: "a2f6fcadc2e047ddb0c111c95420861a" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:34.109054 19364 leader_election.cc:302] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80654f573c764bc18e65d583b751997f, a2f6fcadc2e047ddb0c111c95420861a; no voters:
I20240502 14:15:34.109130 19012 raft_consensus.cc:3032] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:34.109176 19012 raft_consensus.cc:729] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:34.109287 19892 raft_consensus.cc:2781] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:34.109421 19012 consensus_queue.cc:260] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:34.109862 19012 raft_consensus.cc:3037] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:34.113176 19012 raft_consensus.cc:2454] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f6fcadc2e047ddb0c111c95420861a in term 3.
I20240502 14:15:34.115569 19892 raft_consensus.cc:686] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 3 LEADER]: Becoming Leader. State: Replica: a2f6fcadc2e047ddb0c111c95420861a, State: Running, Role: LEADER
I20240502 14:15:34.115960 19892 consensus_queue.cc:237] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:34.120146 18905 catalog_manager.cc:5484] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a reported cstate change: term changed from 2 to 3, leader changed from 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) to a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68). New cstate: current_term: 3 leader_uuid: "a2f6fcadc2e047ddb0c111c95420861a" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:34.453441 19937 tablet_copy_client.cc:523] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:34.456498 19937 tablet_bootstrap.cc:492] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66: Bootstrap starting.
I20240502 14:15:34.480468 19398 raft_consensus.cc:1261] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a [term 4 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 14:15:34.481046 19947 consensus_queue.cc:1035] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:15:34.492800 19088 raft_consensus.cc:1261] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 4 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 14:15:34.493956 19937 tablet_bootstrap.cc:492] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=93 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:34.494244 19946 consensus_queue.cc:1035] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:15:34.494453 19937 tablet_bootstrap.cc:492] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66: Bootstrap complete.
I20240502 14:15:34.494776 19937 ts_tablet_manager.cc:1367] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66: Time spent bootstrapping tablet: real 0.038s user 0.032s sys 0.004s
I20240502 14:15:34.501392 19937 raft_consensus.cc:348] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:34.501582 19937 raft_consensus.cc:729] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: eac995eafe3e4c388a7caf591d6f1c66, State: Initialized, Role: LEARNER
I20240502 14:15:34.501816 19937 consensus_queue.cc:260] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:34.502703 19937 ts_tablet_manager.cc:1398] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:34.503757 19026 tablet_copy_service.cc:342] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Request end of tablet copy session eac995eafe3e4c388a7caf591d6f1c66-99128e8f96dc4e1582ec51cbcb8085e0 received from {username='slave'} at 127.0.0.1:54324
I20240502 14:15:34.503888 19026 tablet_copy_service.cc:433] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: ending tablet copy session eac995eafe3e4c388a7caf591d6f1c66-99128e8f96dc4e1582ec51cbcb8085e0 on tablet 99128e8f96dc4e1582ec51cbcb8085e0 with peer eac995eafe3e4c388a7caf591d6f1c66
I20240502 14:15:34.517328 19088 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 14:15:34.517760 19892 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:34.521071 19012 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 14:15:34.521109 19474 raft_consensus.cc:3037] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66 [term 2 LEARNER]: Advancing to term 3
I20240502 14:15:34.521503 19892 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:34.524999 19474 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66 [term 3 LEARNER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 14:15:34.525631 19892 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:34.534018 19966 raft_consensus.cc:1050] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a: attempting to promote NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 to VOTER
I20240502 14:15:34.534813 19966 consensus_queue.cc:237] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 28, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
I20240502 14:15:34.536830 19012 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:15:34.536830 19474 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66 [term 3 LEARNER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:15:34.537032 19088 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 14:15:34.537457 19892 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:34.537632 19953 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:34.537878 19966 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 14:15:34.539870 19892 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 28 to 30, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:34.541033 19474 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 28 to 30, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:34.543956 19088 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 28 to 30, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:34.544874 18905 catalog_manager.cc:5484] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a reported cstate change: config changed from index 28 to 30, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "a2f6fcadc2e047ddb0c111c95420861a" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:34.546552 19012 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 28 to 30, eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:34.553753 19398 consensus_queue.cc:237] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } }
I20240502 14:15:34.555271 19012 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 14:15:34.555286 19474 raft_consensus.cc:1261] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66 [term 3 FOLLOWER]: Refusing update from remote peer a2f6fcadc2e047ddb0c111c95420861a: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 14:15:34.556136 19892 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:15:34.556752 19953 consensus_queue.cc:1035] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:15:34.558454 19892 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P a2f6fcadc2e047ddb0c111c95420861a [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:34.559084 19474 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:34.559243 19012 raft_consensus.cc:2932] T 99128e8f96dc4e1582ec51cbcb8085e0 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:34.562739 18905 catalog_manager.cc:5484] T 99128e8f96dc4e1582ec51cbcb8085e0 P eac995eafe3e4c388a7caf591d6f1c66 reported cstate change: config changed from index 30 to 31, VOTER 80654f573c764bc18e65d583b751997f (127.15.74.66) evicted. New cstate: current_term: 3 leader_uuid: "a2f6fcadc2e047ddb0c111c95420861a" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } }
I20240502 14:15:34.562638 18961 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99128e8f96dc4e1582ec51cbcb8085e0 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:34.567194 19080 tablet_service.cc:1514] Processing DeleteTablet for tablet 99128e8f96dc4e1582ec51cbcb8085e0 with delete_type TABLET_DATA_TOMBSTONED (TS 80654f573c764bc18e65d583b751997f not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:60996
I20240502 14:15:34.568153 19971 tablet_replica.cc:330] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f: stopping tablet replica
I20240502 14:15:34.568337 19971 raft_consensus.cc:2229] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:34.568647 19971 raft_consensus.cc:2258] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:34.569599 19971 ts_tablet_manager.cc:1875] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:34.573843 19971 ts_tablet_manager.cc:1888] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.30
I20240502 14:15:34.573961 19971 log.cc:1196] T 99128e8f96dc4e1582ec51cbcb8085e0 P 80654f573c764bc18e65d583b751997f: Deleting WAL directory at /tmp/dist-test-taskCZBrv_/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714659223241330-15657-0/minicluster-data/ts-1-root/wals/99128e8f96dc4e1582ec51cbcb8085e0
I20240502 14:15:34.574883 18901 catalog_manager.cc:4830] TS 80654f573c764bc18e65d583b751997f (127.15.74.66:44583): tablet 99128e8f96dc4e1582ec51cbcb8085e0 (table test-workload [id=682791e08deb409e86af13a3a7eb1967]) successfully deleted
I20240502 14:15:35.570370 19549 consensus_queue.cc:237] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:35.572530 19012 raft_consensus.cc:1261] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Refusing update from remote peer d908205271af490dada0a2845191fef4: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
W20240502 14:15:35.572691 19515 consensus_peers.cc:485] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 -> Peer a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68:41443): Couldn't send request to peer a2f6fcadc2e047ddb0c111c95420861a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f260d59158a344beb686ea0f9746f743. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:35.572844 19160 raft_consensus.cc:1261] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Refusing update from remote peer d908205271af490dada0a2845191fef4: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 14:15:35.573089 19973 consensus_queue.cc:1035] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:15:35.573383 19944 consensus_queue.cc:1035] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:15:35.575757 19944 raft_consensus.cc:2932] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 29 to 31, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:35.576509 19160 raft_consensus.cc:2932] T f260d59158a344beb686ea0f9746f743 P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:35.576948 19012 raft_consensus.cc:2932] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:35.581064 18906 catalog_manager.cc:5484] T f260d59158a344beb686ea0f9746f743 P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: config changed from index 29 to 31, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New cstate: current_term: 3 leader_uuid: "d908205271af490dada0a2845191fef4" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:35.583971 19160 consensus_queue.cc:237] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:35.586799 19474 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66 [term 3 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 14:15:35.587170 19012 raft_consensus.cc:1261] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Refusing update from remote peer 4c60ef9c4e8f439facdfdc6fd669ae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 14:15:35.587713 19919 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
W20240502 14:15:35.588017 19131 consensus_peers.cc:485] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 -> Peer d908205271af490dada0a2845191fef4 (127.15.74.70:35089): Couldn't send request to peer d908205271af490dada0a2845191fef4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f465cbf06f4420c8642e8092ce8e1ed. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:35.588264 19919 consensus_queue.cc:1035] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 14:15:35.590101 19982 raft_consensus.cc:2932] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 30 to 31, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:35.590790 19012 raft_consensus.cc:2932] T 8f465cbf06f4420c8642e8092ce8e1ed P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:35.591287 19474 raft_consensus.cc:2932] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:35.595345 18905 catalog_manager.cc:5484] T 8f465cbf06f4420c8642e8092ce8e1ed P eac995eafe3e4c388a7caf591d6f1c66 reported cstate change: config changed from index 30 to 31, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New cstate: current_term: 3 leader_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: false } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:35.598963 19012 consensus_queue.cc:237] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } }
I20240502 14:15:35.601838 19088 raft_consensus.cc:1261] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:15:35.602192 19160 raft_consensus.cc:1261] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Refusing update from remote peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:15:35.602270 19962 consensus_queue.cc:1035] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20240502 14:15:35.602766 18982 consensus_peers.cc:485] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 -> Peer d908205271af490dada0a2845191fef4 (127.15.74.70:35089): Couldn't send request to peer d908205271af490dada0a2845191fef4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 342705e9da8043ae9aebe50310650d2c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:35.602797 19962 consensus_queue.cc:1035] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:35.605048 19987 raft_consensus.cc:2932] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:35.605448 19088 raft_consensus.cc:2932] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:35.606796 19160 raft_consensus.cc:2932] T 342705e9da8043ae9aebe50310650d2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:35.609474 18910 catalog_manager.cc:5484] T 342705e9da8043ae9aebe50310650d2c P 80654f573c764bc18e65d583b751997f reported cstate change: config changed from index -1 to 27, NON_VOTER d908205271af490dada0a2845191fef4 (127.15.74.70) added. New cstate: current_term: 2 leader_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } attrs { replace: true } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "d908205271af490dada0a2845191fef4" member_type: NON_VOTER last_known_addr { host: "127.15.74.70" port: 35089 } attrs { promote: true } } }
I20240502 14:15:35.613718 19088 consensus_queue.cc:237] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 4.28, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } }
I20240502 14:15:35.616348 19012 raft_consensus.cc:1261] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240502 14:15:35.616947 19160 raft_consensus.cc:1261] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
W20240502 14:15:35.616997 19057 consensus_peers.cc:485] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f -> Peer a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68:41443): Couldn't send request to peer a2f6fcadc2e047ddb0c111c95420861a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cb757082b084620af257fb553d2dd4b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:35.617043 19992 consensus_queue.cc:1035] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:35.617455 19915 consensus_queue.cc:1035] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 14:15:35.620015 19995 raft_consensus.cc:2932] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f [term 4 LEADER]: Committing config change with OpId 4.29: config changed from index -1 to 29, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:35.620344 19012 raft_consensus.cc:2932] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Committing config change with OpId 4.29: config changed from index -1 to 29, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:35.620553 19160 raft_consensus.cc:2932] T 1cb757082b084620af257fb553d2dd4b P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 4 FOLLOWER]: Committing config change with OpId 4.29: config changed from index -1 to 29, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:35.624518 18905 catalog_manager.cc:5484] T 1cb757082b084620af257fb553d2dd4b P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 reported cstate change: config changed from index -1 to 29, NON_VOTER a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68) added. New cstate: current_term: 4 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } attrs { replace: true } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: NON_VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: true } } }
I20240502 14:15:35.628069 19088 consensus_queue.cc:237] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } }
I20240502 14:15:35.630641 19012 raft_consensus.cc:1261] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
W20240502 14:15:35.630985 19057 consensus_peers.cc:485] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f -> Peer eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69:44877): Couldn't send request to peer eac995eafe3e4c388a7caf591d6f1c66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 618d11b0b08e4d49ac15d443acd10e2c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:35.631321 19992 consensus_queue.cc:1035] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:35.632122 19160 raft_consensus.cc:1261] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Refusing update from remote peer 80654f573c764bc18e65d583b751997f: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 14:15:35.633003 19992 consensus_queue.cc:1035] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 14:15:35.634023 19995 raft_consensus.cc:2932] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
I20240502 14:15:35.634675 19012 raft_consensus.cc:2932] T 618d11b0b08e4d49ac15d443acd10e2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
I20240502 14:15:35.640121 18906 catalog_manager.cc:5484] T 618d11b0b08e4d49ac15d443acd10e2c P 80654f573c764bc18e65d583b751997f reported cstate change: config changed from index -1 to 27, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New cstate: current_term: 2 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:35.644836 19160 raft_consensus.cc:2932] T 618d11b0b08e4d49ac15d443acd10e2c P 4c60ef9c4e8f439facdfdc6fd669ae78 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER eac995eafe3e4c388a7caf591d6f1c66 (127.15.74.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } attrs { replace: true } } peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "4c60ef9c4e8f439facdfdc6fd669ae78" member_type: VOTER last_known_addr { host: "127.15.74.67" port: 39767 } } peers { permanent_uuid: "eac995eafe3e4c388a7caf591d6f1c66" member_type: NON_VOTER last_known_addr { host: "127.15.74.69" port: 44877 } attrs { promote: true } } }
I20240502 14:15:35.686309 18967 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:15:35.689808 19012 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d"
dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
mode: GRACEFUL
new_leader_uuid: "80654f573c764bc18e65d583b751997f"
from {username='slave'} at 127.0.0.1:54318
I20240502 14:15:35.690029 19012 raft_consensus.cc:595] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 LEADER]: Received request to transfer leadership to TS 80654f573c764bc18e65d583b751997f
I20240502 14:15:35.690462 18967 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 14:15:35.690658 18967 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:15:35.877517 19990 raft_consensus.cc:979] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: : Instructing follower 80654f573c764bc18e65d583b751997f to start an election
I20240502 14:15:35.877705 19990 raft_consensus.cc:1067] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 LEADER]: Signalling peer 80654f573c764bc18e65d583b751997f to start an election
I20240502 14:15:35.878223 19088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d"
dest_uuid: "80654f573c764bc18e65d583b751997f"
from {username='slave'} at 127.0.0.1:60998
I20240502 14:15:35.878427 19088 raft_consensus.cc:482] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:35.878523 19088 raft_consensus.cc:3037] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:35.880997 19088 raft_consensus.cc:504] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:35.881645 19088 leader_election.cc:288] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 5 election: Requested vote from peers 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141), a2f6fcadc2e047ddb0c111c95420861a (127.15.74.68:41443)
I20240502 14:15:35.881950 19012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7"
I20240502 14:15:35.882004 19397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154f396806954d2eb24baa4a86ba876d" candidate_uuid: "80654f573c764bc18e65d583b751997f" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "a2f6fcadc2e047ddb0c111c95420861a"
I20240502 14:15:35.882112 19012 raft_consensus.cc:3032] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:15:35.882169 19012 raft_consensus.cc:729] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1880cd4d2b8e4ec6b42dfd8d2660f5d7, State: Running, Role: LEADER
I20240502 14:15:35.882196 19397 raft_consensus.cc:3037] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:35.882397 19012 consensus_queue.cc:260] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:35.882732 19012 raft_consensus.cc:3037] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:35.884691 19397 raft_consensus.cc:2454] T 154f396806954d2eb24baa4a86ba876d P a2f6fcadc2e047ddb0c111c95420861a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 5.
I20240502 14:15:35.884856 19012 raft_consensus.cc:2454] T 154f396806954d2eb24baa4a86ba876d P 1880cd4d2b8e4ec6b42dfd8d2660f5d7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80654f573c764bc18e65d583b751997f in term 5.
I20240502 14:15:35.885143 19057 leader_election.cc:302] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80654f573c764bc18e65d583b751997f, a2f6fcadc2e047ddb0c111c95420861a; no voters:
I20240502 14:15:35.885411 19992 raft_consensus.cc:2781] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:35.885530 19992 raft_consensus.cc:686] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [term 5 LEADER]: Becoming Leader. State: Replica: 80654f573c764bc18e65d583b751997f, State: Running, Role: LEADER
I20240502 14:15:35.885738 19992 consensus_queue.cc:237] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } }
I20240502 14:15:35.888231 18910 catalog_manager.cc:5484] T 154f396806954d2eb24baa4a86ba876d P 80654f573c764bc18e65d583b751997f reported cstate change: term changed from 4 to 5, leader changed from 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65) to 80654f573c764bc18e65d583b751997f (127.15.74.66). New cstate: current_term: 5 leader_uuid: "80654f573c764bc18e65d583b751997f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "80654f573c764bc18e65d583b751997f" member_type: VOTER last_known_addr { host: "127.15.74.66" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1880cd4d2b8e4ec6b42dfd8d2660f5d7" member_type: VOTER last_known_addr { host: "127.15.74.65" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f6fcadc2e047ddb0c111c95420861a" member_type: VOTER last_known_addr { host: "127.15.74.68" port: 41443 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:35.975169 20001 ts_tablet_manager.cc:902] T 8f465cbf06f4420c8642e8092ce8e1ed P d908205271af490dada0a2845191fef4: Initiating tablet copy from peer 4c60ef9c4e8f439facdfdc6fd669ae78 (127.15.74.67:39767)
I20240502 14:15:35.976724 20001 tablet_copy_client.cc:308] T 8f465cbf06f4420c8642e8092ce8e1ed P d908205271af490dada0a2845191fef4: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.67:39767
I20240502 14:15:35.977272 19174 tablet_copy_service.cc:140] P 4c60ef9c4e8f439facdfdc6fd669ae78: Received BeginTabletCopySession request for tablet 8f465cbf06f4420c8642e8092ce8e1ed from peer d908205271af490dada0a2845191fef4 ({username='slave'} at 127.0.0.1:49510)
I20240502 14:15:35.977416 19174 tablet_copy_service.cc:161] P 4c60ef9c4e8f439facdfdc6fd669ae78: Beginning new tablet copy session on tablet 8f465cbf06f4420c8642e8092ce8e1ed from peer d908205271af490dada0a2845191fef4 at {username='slave'} at 127.0.0.1:49510: session id = d908205271af490dada0a2845191fef4-8f465cbf06f4420c8642e8092ce8e1ed
I20240502 14:15:35.979769 19174 tablet_copy_source_session.cc:205] T 8f465cbf06f4420c8642e8092ce8e1ed P 4c60ef9c4e8f439facdfdc6fd669ae78: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:35.980795 20001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f465cbf06f4420c8642e8092ce8e1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:35.986058 20001 tablet_copy_client.cc:783] T 8f465cbf06f4420c8642e8092ce8e1ed P d908205271af490dada0a2845191fef4: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:35.986335 20001 tablet_copy_client.cc:647] T 8f465cbf06f4420c8642e8092ce8e1ed P d908205271af490dada0a2845191fef4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:35.987713 20005 tablet_copy_client.cc:1002] T 8f465cbf06f4420c8642e8092ce8e1ed P d908205271af490dada0a2845191fef4: tablet copy: Injecting latency into file download: 2000
I20240502 14:15:36.079336 20007 ts_tablet_manager.cc:902] T 1cb757082b084620af257fb553d2dd4b P a2f6fcadc2e047ddb0c111c95420861a: Initiating tablet copy from peer 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:36.081113 20007 tablet_copy_client.cc:308] T 1cb757082b084620af257fb553d2dd4b P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.66:44583
I20240502 14:15:36.082142 19101 tablet_copy_service.cc:140] P 80654f573c764bc18e65d583b751997f: Received BeginTabletCopySession request for tablet 1cb757082b084620af257fb553d2dd4b from peer a2f6fcadc2e047ddb0c111c95420861a ({username='slave'} at 127.0.0.1:32804)
I20240502 14:15:36.082273 19101 tablet_copy_service.cc:161] P 80654f573c764bc18e65d583b751997f: Beginning new tablet copy session on tablet 1cb757082b084620af257fb553d2dd4b from peer a2f6fcadc2e047ddb0c111c95420861a at {username='slave'} at 127.0.0.1:32804: session id = a2f6fcadc2e047ddb0c111c95420861a-1cb757082b084620af257fb553d2dd4b
I20240502 14:15:36.084653 19101 tablet_copy_source_session.cc:205] T 1cb757082b084620af257fb553d2dd4b P 80654f573c764bc18e65d583b751997f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:36.085774 20007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb757082b084620af257fb553d2dd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:36.090660 20007 tablet_copy_client.cc:783] T 1cb757082b084620af257fb553d2dd4b P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:36.090869 20007 tablet_copy_client.cc:647] T 1cb757082b084620af257fb553d2dd4b P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:36.092430 20011 tablet_copy_client.cc:1002] T 1cb757082b084620af257fb553d2dd4b P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Injecting latency into file download: 2000
I20240502 14:15:36.189303 20012 ts_tablet_manager.cc:902] T f260d59158a344beb686ea0f9746f743 P a2f6fcadc2e047ddb0c111c95420861a: Initiating tablet copy from peer d908205271af490dada0a2845191fef4 (127.15.74.70:35089)
I20240502 14:15:36.189900 20013 ts_tablet_manager.cc:902] T 342705e9da8043ae9aebe50310650d2c P d908205271af490dada0a2845191fef4: Initiating tablet copy from peer 1880cd4d2b8e4ec6b42dfd8d2660f5d7 (127.15.74.65:41141)
I20240502 14:15:36.190626 20012 tablet_copy_client.cc:308] T f260d59158a344beb686ea0f9746f743 P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.70:35089
I20240502 14:15:36.191449 19559 tablet_copy_service.cc:140] P d908205271af490dada0a2845191fef4: Received BeginTabletCopySession request for tablet f260d59158a344beb686ea0f9746f743 from peer a2f6fcadc2e047ddb0c111c95420861a ({username='slave'} at 127.0.0.1:49214)
I20240502 14:15:36.191632 19559 tablet_copy_service.cc:161] P d908205271af490dada0a2845191fef4: Beginning new tablet copy session on tablet f260d59158a344beb686ea0f9746f743 from peer a2f6fcadc2e047ddb0c111c95420861a at {username='slave'} at 127.0.0.1:49214: session id = a2f6fcadc2e047ddb0c111c95420861a-f260d59158a344beb686ea0f9746f743
I20240502 14:15:36.195748 19559 tablet_copy_source_session.cc:205] T f260d59158a344beb686ea0f9746f743 P d908205271af490dada0a2845191fef4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:36.196416 20013 tablet_copy_client.cc:308] T 342705e9da8043ae9aebe50310650d2c P d908205271af490dada0a2845191fef4: tablet copy: Beginning tablet copy session from remote peer at address 127.15.74.65:41141
I20240502 14:15:36.197013 19026 tablet_copy_service.cc:140] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Received BeginTabletCopySession request for tablet 342705e9da8043ae9aebe50310650d2c from peer d908205271af490dada0a2845191fef4 ({username='slave'} at 127.0.0.1:54696)
I20240502 14:15:36.197144 19026 tablet_copy_service.cc:161] P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Beginning new tablet copy session on tablet 342705e9da8043ae9aebe50310650d2c from peer d908205271af490dada0a2845191fef4 at {username='slave'} at 127.0.0.1:54696: session id = d908205271af490dada0a2845191fef4-342705e9da8043ae9aebe50310650d2c
I20240502 14:15:36.197520 20012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f260d59158a344beb686ea0f9746f743. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:36.200163 19026 tablet_copy_source_session.cc:205] T 342705e9da8043ae9aebe50310650d2c P 1880cd4d2b8e4ec6b42dfd8d2660f5d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:36.201171 20013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342705e9da8043ae9aebe50310650d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:36.202998 20012 tablet_copy_client.cc:783] T f260d59158a344beb686ea0f9746f743 P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:36.203191 20012 tablet_copy_client.cc:647] T f260d59158a344beb686ea0f9746f743 P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:36.204649 20017 tablet_copy_client.cc:1002] T f260d59158a344beb686ea0f9746f743 P a2f6fcadc2e047ddb0c111c95420861a: tablet copy: Injecting latency into file download: 2000
I20240502 14:15:36.207077 20013 tablet_copy_client.cc:783] T 342705e9da8043ae9aebe50310650d2c P d908205271af490dada0a2845191fef4: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:36.207259 20013 tablet_copy_client.cc:647] T 342705e9da8043ae9aebe50310650d2c P d908205271af490dada0a2845191fef4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:36.208710 20022 ts_tablet_manager.cc:902] T 618d11b0b08e4d49ac15d443acd10e2c P eac995eafe3e4c388a7caf591d6f1c66: Initiating tablet copy from peer 80654f573c764bc18e65d583b751997f (127.15.74.66:44583)
I20240502 14:15:36.209290 20021 tablet_copy_client.cc:1002] T 342705e9da8043ae9aebe50310650d2c P d908205271af490dada0a2845191fef4: tablet copy: Injecting latency into file download: 2000
F20240502 14:15:36.209254 20022 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-154f396806954d2eb24baa4a86ba876d.queue_time_us, tablet-download-pool-154f396806954d2eb24baa4a86ba876d.queue_time_us) (tablet-download-pool-154f396806954d2eb24baa4a86ba876d.run_cpu_time_us, tablet-download-pool-154f396806954d2eb24baa4a86ba876d.run_cpu_time_us) (tablet-download-pool-154f396806954d2eb24baa4a86ba876d.run_wall_time_us, tablet-download-pool-154f396806954d2eb24baa4a86ba876d.run_wall_time_us) (tablet-download-pool-1cb757082b084620af257fb553d2dd4b.queue_time_us, tablet-download-pool-1cb757082b084620af257fb553d2dd4b.queue_time_us) (tablet-download-pool-1cb757082b084620af257fb553d2dd4b.run_cpu_time_us, tablet-download-pool-1cb757082b084620af257fb553d2dd4b.run_cpu_time_us) (tablet-download-pool-1cb757082b084620af257fb553d2dd4b.run_wall_time_us, tablet-download-pool-1cb757082b084620af257fb553d2dd4b.run_wall_time_us) (tablet-download-pool-342705e9da8043ae9aebe50310650d2c.queue_time_us, tablet-download-pool-342705e9da8043ae9aebe50310650d2c.queue_time_us) (tablet-download-pool-342705e9da8043ae9aebe50310650d2c.run_cpu_time_us, tablet-download-pool-342705e9da8043ae9aebe50310650d2c.run_cpu_time_us) (tablet-download-pool-342705e9da8043ae9aebe50310650d2c.run_wall_time_us, tablet-download-pool-342705e9da8043ae9aebe50310650d2c.run_wall_time_us) (tablet-download-pool-35d01415abc44ec4b52577992892c5b1.queue_time_us, tablet-download-pool-35d01415abc44ec4b52577992892c5b1.queue_time_us) (tablet-download-pool-35d01415abc44ec4b52577992892c5b1.run_cpu_time_us, tablet-download-pool-35d01415abc44ec4b52577992892c5b1.run_cpu_time_us) (tablet-download-pool-35d01415abc44ec4b52577992892c5b1.run_wall_time_us, tablet-download-pool-35d01415abc44ec4b52577992892c5b1.run_wall_time_us) (tablet-download-pool-567ee47202bd46048e318b4dbbf3ff03.queue_time_us, tablet-download-pool-567ee47202bd46048e318b4dbbf3ff03.queue_time_us) (tablet-download-pool-567ee47202bd46048e318b4dbbf3ff03.run_cpu_time_us, tablet-download-pool-567ee47202bd46048e318b4dbbf3ff03.run_cpu_time_us) (tablet-download-pool-567ee47202bd46048e318b4dbbf3ff03.run_wall_time_us, tablet-download-pool-567ee47202bd46048e318b4dbbf3ff03.run_wall_time_us) (tablet-download-pool-618d11b0b08e4d49ac15d443acd10e2c.queue_time_us, tablet-download-pool-618d11b0b08e4d49ac15d443acd10e2c.queue_time_us) (tablet-download-pool-6ef2dcb2456949cfa34b5c2e8b0d2e6d.queue_time_us, tablet-download-pool-6ef2dcb2456949cfa34b5c2e8b0d2e6d.queue_time_us) (tablet-download-pool-6ef2dcb2456949cfa34b5c2e8b0d2e6d.run_cpu_time_us, tablet-download-pool-6ef2dcb2456949cfa34b5c2e8b0d2e6d.run_cpu_time_us) (tablet-download-pool-6ef2dcb2456949cfa34b5c2e8b0d2e6d.run_wall_time_us, tablet-download-pool-6ef2dcb2456949cfa34b5c2e8b0d2e6d.run_wall_time_us) (tablet-download-pool-73c203a4764245dba0944537c7d10129.queue_time_us, tablet-download-pool-73c203a4764245dba0944537c7d10129.queue_time_us) (tablet-download-pool-73c203a4764245dba0944537c7d10129.run_cpu_time_us, tablet-download-pool-73c203a4764245dba0944537c7d10129.run_cpu_time_us) (tablet-download-pool-73c203a4764245dba0944537c7d10129.run_wall_time_us, tablet-download-pool-73c203a4764245dba0944537c7d10129.run_wall_time_us) (tablet-download-pool-76a8747fc61249759eb84a0110df4efb.queue_time_us, tablet-download-pool-76a8747fc61249759eb84a0110df4efb.queue_time_us) (tablet-download-pool-76a8747fc61249759eb84a0110df4efb.run_cpu_time_us, tablet-download-pool-76a8747fc61249759eb84a0110df4efb.run_cpu_time_us) (tablet-download-pool-76a8747fc61249759eb84a0110df4efb.run_wall_time_us, tablet-download-pool-76a8747fc61249759eb84a0110df4efb.run_wall_time_us) (tablet-download-pool-8f465cbf06f4420c8642e8092ce8e1ed.queue_time_us, tablet-download-pool-8f465cbf06f4420c8642e8092ce8e1ed.queue_time_us) (tablet-download-pool-8f465cbf06f4420c8642e8092ce8e1ed.run_cpu_time_us, tablet-download-pool-8f465cbf06f4420c8642e8092ce8e1ed.run_cpu_time_us) (tablet-download-pool-8f465cbf06f4420c8642e8092ce8e1ed.run_wall_time_us, tablet-download-pool-8f465cbf06f4420c8642e8092ce8e1ed.run_wall_time_us) (tablet-download-pool-99128e8f96dc4e1582ec51cbcb8085e0.queue_time_us, tablet-download-pool-99128e8f96dc4e1582ec51cbcb8085e0.queue_time_us) (tablet-download-pool-99128e8f96dc4e1582ec51cbcb8085e0.run_cpu_time_us, tablet-download-pool-99128e8f96dc4e1582ec51cbcb8085e0.run_cpu_time_us) (tablet-download-pool-99128e8f96dc4e1582ec51cbcb8085e0.run_wall_time_us, tablet-download-pool-99128e8f96dc4e1582ec51cbcb8085e0.run_wall_time_us) (tablet-download-pool-ac2b64044a444b968b6e895784126f1b.queue_time_us, tablet-download-pool-ac2b64044a444b968b6e895784126f1b.queue_time_us) (tablet-download-pool-ac2b64044a444b968b6e895784126f1b.run_cpu_time_us, tablet-download-pool-ac2b64044a444b968b6e895784126f1b.run_cpu_time_us) (tablet-download-pool-ac2b64044a444b968b6e895784126f1b.run_wall_time_us, tablet-download-pool-ac2b64044a444b968b6e895784126f1b.run_wall_time_us) (tablet-download-pool-f260d59158a344beb686ea0f9746f743.queue_time_us, tablet-download-pool-f260d59158a344beb686ea0f9746f743.queue_time_us) (tablet-download-pool-f260d59158a344beb686ea0f9746f743.run_cpu_time_us, tablet-download-pool-f260d59158a344beb686ea0f9746f743.run_cpu_time_us) (tablet-download-pool-f260d59158a344beb686ea0f9746f743.run_wall_time_us, tablet-download-pool-f260d59158a344beb686ea0f9746f743.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714659336 (unix time) try "date -d @1714659336" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800003d29) received by PID 15657 (TID 0x7f27edcb8700) from PID 15657; stack trace: ***
@ 0x7f28a8eaf980 (unknown) at ??:0
@ 0x7f289e1fcfb7 gsignal at ??:0
@ 0x7f289e1fe921 abort at ??:0
@ 0x7f28a0cb1dcd google::LogMessage::Fail() at ??:0
@ 0x7f28a0cb5b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f28a0cb17cc google::LogMessage::Flush() at ??:0
@ 0x7f28a0cb2f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f28a232e742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f28a22faf8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f28a22f4cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f28b17c08de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f28b17d8daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f28b1903480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f28b193ba8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f28b191e981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f28b191e70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f28b67c098d std::function<>::operator()() at ??:0
@ 0x7f28a22feb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f28a230219c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f28a2302019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f28b67c098d std::function<>::operator()() at ??:0
@ 0x7f28a22dd01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f28a8ea46db start_thread at ??:0
@ 0x7f289e2df71f clone at ??:0